diff --git a/.gitignore b/.gitignore index 4790a19..314803d 100644 --- a/.gitignore +++ b/.gitignore @@ -67,3 +67,6 @@ scripts/deploy_r740.secrets # git worktrees (multi-instance dev) .trees/ + +# sudo run-script output — the scripts are committed, their logs are not +scripts/sudo/logs/ diff --git a/README_FOR_GPS3_CLAUDE.md b/README_FOR_GPS3_CLAUDE.md new file mode 100644 index 0000000..6f11a25 --- /dev/null +++ b/README_FOR_GPS3_CLAUDE.md @@ -0,0 +1,274 @@ +# READ ME FIRST — instructions for the Claude Code session on gps3 + +**Written 2026-08-04 by the Claude Code session on the T420** (user `finch`), +immediately before this drive was unplugged and carried to the R740. + +You are reading this because the drive is now plugged into the server's USB +port. This file is your entry point. **Read all of it before touching +anything** — this drive is a single point of failure for the project, and §1 +explains why. + +--- + +## 0. What this drive is + +Label `DOSTB20150918`, 1.9 TB, **NTFS**. It is the user's personal drive, and +after the 2026-07 evacuation of the failing Backup Plus drive it became the +**only complete copy of the PHIVOLCS legacy GNSS archive**. + +Two separate things are on it that concern you: + +| Path | Size | What | +|---|---|---| +| `processing_files/` | **22 GB** | Bernese **5.2** LUZON processed set from Abegail — the new work (§3) | +| `RECOVERED_*/` (3 dirs) | **26 GB** | the legacy archive rescued from three dead/dying drives (§4) | + +Also present and **none of your business**: `ps4e/` (99 GB InSAR pipeline from +prior paid work — valuable, never suggest deleting it), plus personal media, +`Pictures`, `Books`, `Music`, `Obsidian`. **Do not scan, index, catalogue or +copy anything outside the two paths above.** This is a personal drive that +happens to carry work data. + +--- + +## 1. ⚠ Before you run a single command + +**Mount it read-only.** On the T420 it was mounted `ro` and that is how it +should stay: + +```bash +sudo mkdir -p /mnt/dostb +sudo mount -o ro,noatime /dev/sdX1 /mnt/dostb # confirm sdX1 with lsblk -f first +``` + +Three reasons, in order of seriousness: + +1. **It is the only complete copy of the legacy archive.** `/srv/gnss-archive/legacy` + on this machine is still empty (your own §13.6). Until a second copy exists, + a write error here is unrecoverable data loss for the project. +2. **The source drives are dead or dying.** The `RECOVERED_*` directories were + rescued from failed hardware. There is nothing to re-rescue from. +3. **NTFS-3G write support is fine but not free.** A read-only mount removes + the entire class of accident. + +**Confirm the device letter with `lsblk -f` every time.** It is not stable +across reboots or other USB devices, and this project has a documented history +of `sdX` shifting between sessions. + +--- + +## 2. The opportunity this drive being here creates + +**This unblocks §13.6 of your own session log.** That entry records +`/srv/gnss-archive/legacy` as empty, with the transfer *"blocked on the push +from the T420's DOSTB mount"* — over wifi measured at **6 MB/s**. + +The drive is now on your USB bus. That bottleneck is gone. 26 GB of +`RECOVERED_*` at USB speed is minutes, not a day. + +**This is the single highest-value thing you can do while the drive is +attached**, and it is worth more than the Bernese work in §3. It takes the +archive from one copy on failing-media-derived personal hardware to two copies +on independent hardware, which is item 5 of the continuity audit. + +Suggested, but **confirm with the user before starting** — it is their drive +and their call: + +```bash +tmux new -As archive # FIRST. A process cannot be moved into tmux later. +rsync -aHAX --info=progress2 /mnt/dostb/RECOVERED_HD-LBU2_WD20EARS_WCAZA4430660/ \ + /srv/gnss-archive/legacy/RECOVERED_HD-LBU2_WD20EARS_WCAZA4430660/ +# repeat per RECOVERED_* directory +``` + +Then **verify with a census, not with rsync's exit code.** Your §13.6 already +records why: rsync exits 0 having skipped files it could not read, and exits 23 +on a run that copied 99.99% successfully. `/srv/gnss-archive/verify_archive.sh` +is staged for this — count files, symlinks, directories and bytes *separately* +and compare against the same census taken on the source. + +**Expect zero symlinks on the source side.** The drive is NTFS; a `find` across +`processing_files/` returned 0 symlinks in 67,553 files. That is a property of +the source, not a transfer failure — do not chase it as a bug. + +--- + +## 3. The Bernese 5.2 set — what it is for + +`processing_files/` is a complete processed run of the **LUZON** network from +**Abegail** (she runs that network at PHIVOLCS), produced under **Bernese 5.2**. +The user's goal, in his words: *"i-reprocess ko sa GPS3 under Bernese 5.4 for +comparison ng results / adjustment (fine tuning) ng PCF."* + +### Already captured into git — do not re-derive + +The **configuration** half is committed to the repo, so you can work from +version control rather than from this drive: + +``` +config/bernese/gpsuser52-luzon/ PHIVOL_REL.PCF (+ PAGENET/PAGENET2 ancestors), + 11 OPT dirs, LUZON.{STA,CRD,VEL,ABB,CLU,BLQ,ATL,PLD} +config/bernese/gpsuser/ the 5.4 PAGENET counterpart from the T420 +``` + +**Read `config/bernese/gpsuser52-luzon/PROVENANCE.md` before using any of it.** +It records the hazards. Two matter most: + +- **Six live `.INP` panels carry `C:\Bernese\…` absolute paths.** On Linux a + backslash is a literal character, not a separator, so these do not fail + loudly. Run `scripts/provision_gpsuser.py` over the tree first. +- **The OPT tree is not uniformly a LUZON set.** Of the live `.INP` panels, 38 + name campaign `${P}/PHIVOLCS`, 9 name `LUZON`, 2 name `SAMR_LYT`, 1 `EXAMPLE`. + These are last-saved menu states from different work, not a curated + gold standard. The BPE sets the campaign at runtime so this is mostly + cosmetic — but do not assume the directory name describes the contents. + +### The data half stays here + +| Path on this drive | Contents | +|---|---| +| `processing_files/GPSDATA/DATAPOOL/LUZON/` | **741 RINEX obs, DOY 121–151 of 2025** (31 days), 25 stations | +| `processing_files/GPSDATA/CAMPAIGN/LUZON/` | OBS 2064, ORB 554, ATM 154, **SOL 1944**, OUT 6582 | +| `processing_files/GPSDATA/CAMPAIGN/LUZON/SOL/` | **the comparison target** — `F1_*.NQ0` dailies, `WK_2413`/`WK_2414` `.NQ0`/`.SNX` | +| `processing_files/DATAPOOL_IGS/` | 94 SP3 | +| `processing_files/DATAPOOL_BSW52/` | 39 ION | +| `processing_files/BERN52/GPS/GEN/` | `C04_*.ERP` from 1986, `BULLET_A.ERP`, `ANT_COD_I14.PCV`, `ANT_COD_I20*.PCV` | + +`CAMPAIGN/LUZON/RAW`, `ORX`, `GRD` are **empty and that is fine** — `OBS` holds +the converted observations. Not a truncated delivery. + +### ⚠ The comparison is invalid unless you control the models + +This is the part most likely to go wrong, because the failure produces a +plausible number rather than an error. + +| | 5.2 LUZON (hers) | 5.4 PAGENET (this machine) | +|---|---|---| +| Antenna / PCV | **I14** | **I20** | +| GNSS for ambiguity resolution | **ALL** | **GRE** | +| Baselines, `V_CLU` | 6000/2000/200/20, 10 | identical | + +The tuning parameters agree. The **models do not.** I14 → I20 is a cm-level, +largely vertical, systematic shift — comfortably inside the range a PCF tuning +change could produce. + +**So: reprocess under I14 first and reproduce her numbers.** Only once you can +land on her `SOL/` values should you vary anything. A fresh I20 run compared +against her I14 result would show a difference that is real, reproducible, and +has nothing to do with the PCF. + +The `ANT_COD_I14.PCV` you need is on this drive at `BERN52/GPS/GEN/`. + +--- + +## 4. The `RECOVERED_*` directories + +| Directory | Size | Origin | +|---|---|---| +| `RECOVERED_SEAGATE_W2A0W9T2_DATA0` | **~131 G (inferred)** | **omitted from the original table — see below** | +| `RECOVERED_HD-LBU2_WD20EARS_WCAZA4430660` | 14 G | a dead Buffalo external | +| `RECOVERED_DOSTB20150918_from_BackupPlus` | 9.0 G | rescued off the failing Backup Plus | +| `RECOVERED_GPS_1TB_2_WD10EARS_WCAV5M032380` | 3.0 G | a dead 1 TB WD | + +> ### ⚠ Corrected 2026-08-04 — the original table was missing a directory +> +> This section originally listed **three** directories totalling 26 GB and stated +> that this was "not the ~157 GB the continuity audit refers to". The user +> identified a **fourth**, `RECOVERED_SEAGATE_W2A0W9T2_DATA0`, absent from the +> table entirely. The arithmetic supports him: +> +> | | | +> |---|---| +> | The three originally listed | 26 G | +> | `RESUME_NEXT.md`, **measured**: "DOSTB 157 G (RAW alone 125 G)" | 157 G | +> | Therefore the omitted directory | **~131 G** | +> +> That gap is close to the 125 G of RAW the continuity audit calls out +> separately, so `RECOVERED_SEAGATE_*` most likely holds the bulk raw +> observations — **the most valuable and least reproducible material on the +> drive.** The one directory left out of the transfer plan was the one that +> mattered most. +> +> **Consequences for the plan:** +> - "26 GB — minutes, not a day" understates it. ~157 G over USB 3 is roughly +> 30–60 minutes at 50–100 MB/s; still nothing like the 6 MB/s wifi path, but +> not incidental. Do not start it five minutes before the drive must come off. +> - Copying **all four** brings the on-server copy to the full 157 G the audit +> describes, which materially changes what can honestly be claimed afterwards. +> - **Never enumerate these by a typed list again.** The omission happened +> because three names were written out by hand. `scripts/sudo/archive_transfer.sh` +> globs `RECOVERED_*` and reports what it finds, so a fifth directory would be +> picked up rather than silently skipped. +> +> Sizes above remain **unverified on this machine** — the 131 G is inferred from +> subtraction, not measured. The census script reports actual figures once the +> drive is mounted, and those supersede this table. + +Do not report "the archive is backed up" after copying these — report exactly +what you copied, by directory and by census. + +They have never been checksummed. A `sha256sum` manifest written into +`/srv/gnss-archive/manifests/` **and committed to git** is item 4 of the +continuity audit and costs nothing while the drive is attached. Fingerprints +stored only beside the data prove nothing if that disk is what failed. + +--- + +## 5. House rules that apply to you here + +From `docs/gps3-sessions/SESSION_LOG_20260729_storage.md` and +`docs/GPS3_COORDINATION_ONBOARDING.md` — all earned the hard way: + +- **`tmux` first, always.** A running process cannot be moved into it + afterwards. A multi-GB rsync outside tmux dies with the connection. +- **Do not run a bulk transfer and a BPE at the same time.** Headless BPE hangs + under concurrent heavy I/O. Check with `pgrep -af 'rnx2snx|RUNBPE'` — and + note `pgrep -f` matches its own command line, which has produced false + readings three times on this machine. +- **Exit codes lie in both directions here.** `rsync` 0 on skipped files; + `lsof +D` 1 on a clean tree; `grep -c error` returning 3 on a successful BPE + run; `slow_cmd | grep -q` returning 141 under `pipefail`. Verify the effect, + not the status. +- **A check that reports success without having inspected anything** is this + project's recurring defect — your own §15.5 names five instances. A census + that counts zero files and passes is the same bug. +- **Everything reaches `main` through a PR** (Rule 1). Branches live at most a + week. + +--- + +## 6. What the T420 session did today, so you do not repeat it + +- Committed **`PAGENET_DLY.PCF` + `PGN_WK` panels** from the T420 → + `config/bernese/gpsuser/`. This closes your §15.3 blocker. Note the finding: + **that PCF has no dangling WAIT** — §14.5 expected `599` to wait on an + undefined `522`, but it waits on `512 514` and there is no 521/522 in the + file. It is a deliberate reduction, not an unsafe truncation. **PR #65.** +- Committed the **5.2 LUZON configuration** → `config/bernese/gpsuser52-luzon/` + (same PR). `PHIVOL_REL.PCF` (84 PIDs, dated 2025-09-12) contains the + `521/522` R2S_RED branch, `530 ADD_WK`/`531 ADD_MON`, and the + `901/902/903/991` save-summarise-clean tail that `PAGENET_DLY` lacks — so + **readiness item M now has a reference implementation.** +- Corrected the R740 hardware figures in a sibling handover after your §14.2 + measured **12 physical cores / 62 GB**, not the 24 two documents assumed. + +**Check whether PRs #61–#65 have merged before assuming `main` has any of +this.** As of writing, `origin/main` was still at `1d1082e` (PR #60) while +your §15.4 stated all work had landed — it had not. Verify `origin/main` +actually advanced rather than trusting a command's exit code; that is Rule 5, +and this is the case it was written for. + +--- + +## 7. Priority order, if you want one + +1. **Confirm with the user, then copy `RECOVERED_*` to `/srv/gnss-archive/legacy/`.** + Time-boxed by how long the drive is attached. Nothing else on this list is + irreversible if skipped; this one is. +2. **`sha256sum` manifest** of what landed, into git. +3. **Copy the LUZON data** needed for the reprocessing into `$P`/`$D`. +4. **Reprocess under I14** and reproduce Abegail's `SOL/` numbers. +5. Only then vary models or tune the PCF. + +If the drive has to come off before 1 is done, say so plainly and say what did +not get copied. An honest partial is worth more than a hopeful summary. diff --git a/config/bernese/gpsuser/README.md b/config/bernese/gpsuser/README.md new file mode 100644 index 0000000..d02b94a --- /dev/null +++ b/config/bernese/gpsuser/README.md @@ -0,0 +1,118 @@ +# Gold-standard `$U` (GPSUSER) configuration + +This tree is the **source of truth** for everything PHIVOLCS-specific in the +Bernese user directory. `scripts/provision_gpsuser.py` copies it into `$U` +(`~/GPSUSER`), sanitizing on the way. Readiness task **P1-H**. + +**Never hand-edit `$U` directly.** Edit here, commit, re-provision. That is the +whole point of the tree existing. + +--- + +## Why this exists + +Two reasons, and the second is the operational one. + +**Reproducibility.** The MIS team reconfigures the production server. When they +do, a working Bernese environment has to be recoverable by re-running +provisioning — not by someone re-debugging panels by hand from memory. Nothing +here should be recoverable *only* from a running machine. + +**Panels do not survive travel.** Every panel that came off a Windows Bernese +install carried problems into the Linux run: `\` path separators taken as +literal characters, a dangling `WAIT=522` that would have hung the BPE on a PID +that no longer existed, and hardcoded session/campaign literals from the +instructor's demo week. The provisioner catches all three classes on the way +out, so they cannot reach `$U` again. + +--- + +## Layout + +Mirrors `$U`'s own structure. Anything placed here lands at the same relative +path under `~/GPSUSER`. + +``` +SCRIPT/ Perl drivers. Copied VERBATIM — a backslash in Perl is an escape, + not a path separator, so separator conversion would corrupt them. +PCF/ Process Control Files. Checked for dangling WAIT references + before copying; a dangling WAIT blocks the BPE forever. +OPT/ Panel directories (*.INP). Separator-sanitized, and ADDNEQ2.INP + gets MAXPAR sized from the station count. +``` + +### What is deliberately NOT here + +**`PAN/USER.CPU`** — host-specific. `maxjobs` must track the *physical* core +count of whichever machine is running (the BPE sub-solves are FPU-bound, so +hyperthreads do not help and oversubscription is slower than a correct value). +Versioning it would carry one machine's core count onto another — exactly the +bug found on gps3 on 2026-08-03, where `USER.CPU` still read `maxjobs 2` from +the T420 and the 12-core R740 was using two of its cores. The provisioner +**generates** this file from the detected hardware instead. + +**Stock Bernese panels and PCFs.** `$U/OPT` and `$U/PCF` on gps3 are currently +byte-identical to the `$C/USER` template shipped with Bernese 5.4. Vendor files +belong to the vendor; committing 47 unmodified panel directories would bury the +handful of files we actually own. Only *our* additions and deltas go here. + +--- + +## STATUS — incomplete, and what is missing + +`scripts/run_pagenet_week.sh` requires three things in `$U`: + +| Asset | State | +|---|---| +| `SCRIPT/pagenet_pcs.pl` | **Here.** Stock `rnx2snx_pcs.pl` with `PCF_FILE` / `BPE_CAMPAIGN` parameterized. | +| `PAN/USER.CPU` | Generated by the provisioner from detected cores. | +| `PCF/PAGENET_DLY.PCF` | **MISSING — blocks the acceptance test.** | + +### `PAGENET_DLY.PCF` must be captured from the T420 + +It exists **only** on the T420, where it drove the full 7-day training week +unattended. It is not in this repo and not on gps3. + +**Do not re-derive it.** It is described as RNX2SNX modules 1–14 (PID 001→514), +but that is not a truncation you can perform safely by eye: + +- `599 DUMMY` waits on `512 514 522`. Dropping the R2S_RED branch (521/522) + leaves `599` waiting on a PID that never runs — the BPE blocks forever. This + is precisely the dangling-WAIT class the provisioner now refuses. +- The `9xx` save/cleanup tail (`R2S_SUM`, `R2S_SAV`, `OBS_SAV`, `R2S_DEL`, + `BPE_CLN`) needs deliberate decisions about what is kept, not defaults. +- `V_CLUFIN` clustering interacts with the final-solution step and is the other + half of the 502 GPSCLU_P bottleneck fix. + +A re-derived PCF would be a *different* PCF from the one that was validated, and +the acceptance test would then be testing something nobody has ever run. + +**To capture it,** from the T420: + +```bash +cp "$U/PCF/PAGENET_DLY.PCF" /config/bernese/gpsuser/PCF/ +cp -r "$U/OPT/PGN_WK" /config/bernese/gpsuser/OPT/ # if present +``` + +then commit, and on gps3 pull and run the provisioner. `OPT/PGN_WK/ADDNEQ2.INP` +is known to carry Windows separators, a dangling `WAIT=522`, and hardcoded +sessions (`20261030/40/50` — the instructor's demo week); the provisioner fixes +the first, refuses on the third, and reports the second. Expect it to reject the +panel on first attempt. **That is the tool working**, not a malfunction — remap +the hardcoded literals, then re-run. + +--- + +## Usage + +```bash +# Show what would change. Default — writes nothing. +uv run python scripts/provision_gpsuser.py + +# Apply, sizing ADDNEQ2 MAXPAR for a 72-station network +uv run python scripts/provision_gpsuser.py --apply --stations 72 +``` + +The provisioner is **strict by default**: a panel carrying a hazard it cannot +safely auto-fix aborts the whole run before anything is written, so `$U` is +never left half-updated. diff --git a/config/bernese/gpsuser/SCRIPT/pagenet_pcs.pl b/config/bernese/gpsuser/SCRIPT/pagenet_pcs.pl new file mode 100755 index 0000000..c8f5202 --- /dev/null +++ b/config/bernese/gpsuser/SCRIPT/pagenet_pcs.pl @@ -0,0 +1,71 @@ +#!/usr/bin/env perl + +# ============================================================================ +# +# Name : rnx2snx_pcs.pl +# +# Purpose : Start PAGENET BPE process for a particular session +# +# Author : R. Dach +# Created : 07-Jun-2022 +# +# Changes : 07-Jun-2022 RD: Created for version 5.4 +# +# ============================================================================ +use strict; + +use lib $ENV{BPE}; +use startBPE; +use bpe_util; + +# Check arguments +# --------------- +if (@ARGV < 2 or @ARGV > 3 or lc($ARGV[0]) eq "-h") { + die "\n Start PAGENET BPE process for a particular session\n". + "\n Usage: pagenet_pcs.pl [-h] yyyy ssss [pcf]\n". + "\n yyyy : 4-digit (or 2-digit) year". + "\n ssss : 4-character session". + "\n pcf : PCF name (default PAGENET; use PAGENET_DLY for daily 001-514)". + "\n -h : Display this help text\n\n" } + +my $pcf = $ARGV[2] || "PAGENET"; + +# Create startBPE object +# ---------------------- +my $bpe = new startBPE(); + +# Redefine mandatory variables +# ---------------------------- +$$bpe{PCF_FILE} = $pcf; +$$bpe{CPU_FILE} = "USER"; +$$bpe{BPE_CAMPAIGN} = "PAGENET"; +$$bpe{YEAR} = $ARGV[0]; +$$bpe{SESSION} = $ARGV[1]; +$$bpe{SYSOUT} = $pcf; +$$bpe{STATUS} = "$pcf.RUN"; +$$bpe{TASKID} = "RS"; + +# Reset CPU file +# -------------- +$bpe->resetCPU(); + +# Start BPE process +# ----------------- +print "\nPAGENET BPE process started on ".timstr(localtime(time))."\n"; + +# The BPE runs +# ------------ +$bpe->run(); + +# Check for error +# --------------- +if ($$bpe{ERROR_STATUS} ) { + die ("Error in PAGENET BPE: $$bpe{PCF_FILE}.PCF (Session: $ARGV[1])\n"); +} + +# BPE process finished +# -------------------- +print "PAGENET BPE process finished on ".timstr(localtime(time))."\n\n"; + +__END__ + diff --git a/docs/bernese_orchestration_explainer.md b/docs/bernese_orchestration_explainer.md index d87139c..7a2e342 100644 --- a/docs/bernese_orchestration_explainer.md +++ b/docs/bernese_orchestration_explainer.md @@ -1,5 +1,13 @@ # GNSS Pipeline Orchestration: What We're Building for Our Processing Workflow +**Drafted:** 2026-02-27 **Revised:** 2026-08-03 + +> **What has changed since this was written.** In February this described a plan. +> Since then the NAMRIA training week (June) ran the whole PAGENET pipeline +> unattended on live data, and the Dell R740 (§4 below) now has Bernese 5.4 +> installed and verified. Sections marked **[now real]** describe things that +> exist; the rest is still ahead. Timings have been replaced with measured ones. + --- ## The Problem We All Know @@ -15,7 +23,7 @@ Every GNSS processing cycle involves the same invisible overhead: Edit it manually. Try to remember the exact column widths. - Run BPE. Watch the screen. Something failed at step 221 (RXOBV3) — station header mismatch. Find the bad station. Fix the `.STA` entry. Re-run from scratch. -- Wait 35 minutes. Check `RNX2SNX.OUT`. Everything looks OK. +- Wait out the run. Check `RNX2SNX.OUT`. Everything looks OK. Copy the SINEX files to SAVEDISK. Update the tracking spreadsheet. Multiply this by every session, every campaign, every year. Then ask: **how much of this work is science, and how much is file management?** @@ -31,7 +39,7 @@ It is a **conductor** — a program that knows the correct sequence of steps, pe For our workflow, one orchestrated processing run looks like this: ``` -You specify: Campaign = PIVSMIND, Year = 2023, Session = 0100 +You specify: Campaign = PAGENET, Year = 2026, Session = 0860 Orchestrator: 1. Downloads IGS precise orbits + clocks from CDDIS (with automatic fallback to IGN/BKG) @@ -39,7 +47,7 @@ Orchestrator: 3. Validates RINEX files against the station info (.STA) — flags mismatches BEFORE Bernese 4. Decompresses Hatanaka files (CRX2RNX) and copies to RAW/ 5. Renders the RUNBPE.INP and OPT_DIR panel files for this specific campaign - 6. Calls Bernese non-interactively: runs all 47 BPE steps + 6. Calls Bernese non-interactively: runs every BPE step in the PCF 7. Checks the output: station count after RXOBV3, ambiguity fixing rate, HELMCHK residuals 8. Extracts SINEX coordinates, converts to ENU, stores in the database 9. Sends a summary report: what ran, what succeeded, what needs attention @@ -47,13 +55,13 @@ Orchestrator: You receive: A report. Processed results in the database. Any exceptions flagged for your review. ``` -The 47 BPE steps — RNXGRA, RXOBV3, MAUPRP, GNSQIF, HELMCHK, all of them — run exactly as they do today, with exactly the same Bernese software, the same PCF, the same INP file settings. **The science does not change.** What changes is who carries the files. +Every BPE step — RNXGRA, RXOBV3, MAUPRP, GNSQIF, HELMCHK, all of them — runs exactly as it does today, with exactly the same Bernese software, the same PCF, the same INP file settings. **The science does not change.** What changes is who carries the files. --- ## Why This Matters: Three Concrete Pain Points -### 1. The Reproducibility Problem +### 1. The Reproducibility Problem **[partly real]** If you processed session 2023/010 in February, and re-process it in December with a newer IGS final orbit, **will you get the same answer?** Currently, probably not — and you may not be able to explain why, because the exact settings used in February are not recorded anywhere. They lived in a panel file that has since been overwritten. @@ -64,7 +72,14 @@ With orchestration, every processing run produces a complete record: - Which stations survived RXOBV3 and which were dropped - What the HELMCHK residuals were -This is the difference between a result you can publish and a result you can only use internally. +This is the difference between a result you can publish and a result you can only +use internally. + +**Where this stands:** the settings side is now version-controlled — panels, PCFs +and drivers live in the repository and are applied to the server by one command, +so "which settings were active" is answerable from the commit history rather than +from memory. The per-run record of *outcomes* (which stations survived, what the +residuals were) is still to come. ### 2. The Knowledge Concentration Problem @@ -78,11 +93,21 @@ PHIVOLCS operates approximately 270 active stations nationwide as of December 20 An orchestrated pipeline processes all stations in parallel, overnight, every night, without anyone sitting at a terminal. The processing staff review the exception report in the morning — they spend their time on the results that need judgment, not on the steps that don't. -### 4. The Workstation Problem +### 4. The Workstation Problem **[now real]** -A BPE run occupies whatever machine it runs on. Right now, that means a desktop is tied up for 35 minutes — longer for multi-session batches — while you wait, watch, or find something else to do on a different machine. +A BPE run occupies whatever machine it runs on, and it is not a coffee break. Measured: the 54-station EXAMPLE campaign takes **11 minutes**, but a real **72-station PAGENET day took about 2 hours** on the T420 — roughly 40 minutes of it inside a single step (PID 502, GPSCLU_P) solving the final system on one core. Multiply by seven days of a processing week. -The orchestrated pipeline runs on the dedicated Dell server (R740). You submit a processing job from your desk, and your workstation is immediately free. The server handles the computation; you receive the results. Your desktop is no longer a processing node. +The orchestrated pipeline runs on the dedicated Dell server (R740). You submit a +processing job from your desk, and your workstation is immediately free. The server +handles the computation; you receive the results. Your desktop is no longer a +processing node. + +**This part now exists.** Bernese 5.4 was installed on the R740 on 2026-07-29 and +verified against the reference solution to **0.0000 mm** — the same numbers as the +laptop, on twelve cores instead of two, with the campaign data on a dedicated 4 TB +volume. Jobs can also be supervised remotely: a run started at the office was +driven from a home network, with no terminal left open and nothing exposed to the +internet. --- @@ -94,7 +119,7 @@ Orchestration automates the mechanical steps. The judgment steps remain human: |-----------|-------| | IGS product download and staging | Deciding which IGS product tier to use (ultra-rapid vs. rapid vs. final) | | RINEX decompression and file staging | Reviewing HELMCHK flags for possible co-seismic displacement events | -| BPE execution (all 47 steps) | Interpreting anomalies in the ambiguity fixing rate | +| BPE execution (every step in the PCF) | Interpreting anomalies in the ambiguity fixing rate | | RXOBV3 station drop detection | Deciding whether a dropped station reflects a real data problem | | SINEX coordinate extraction | Velocity model review and publication | | Daily ENU coordinate storage | Offset event classification (EQ, equipment change, unknown) | @@ -113,7 +138,7 @@ What the browser tool adds that the current script does not: automatic pre-flagg **Before orchestration:** -> Staff member spends 2–3 hours per campaign session on file management, downloads, and manual BPE setup. BPE runs ~35 minutes attended. Post-processing (SINEX extraction, spreadsheet update) takes another hour. Outlier review requires running a separate Windows script per station, right-clicking bad points, then manually editing PLOT files. One person's full day is consumed by a single session. Errors from manual steps (wrong `.STA` entry, stale orbit file, missed outlier epoch) are caught late. +> Staff member spends 2–3 hours per campaign session on file management, downloads, and manual BPE setup. A real PAGENET day runs ~2 hours attended. Post-processing (SINEX extraction, spreadsheet update) takes another hour. Outlier review requires running a separate Windows script per station, right-clicking bad points, then manually editing PLOT files. One person's full day is consumed by a single session. Errors from manual steps (wrong `.STA` entry, stale orbit file, missed outlier epoch) are caught late. **After orchestration:** @@ -138,13 +163,45 @@ The INP file settings — the GPSEST ionosphere configuration, the MAUPRP cycle ## What We Need From You -To build the templates correctly, we need copies of the INP files from a working production run: - -1. The **`R2S_GEN` subdirectory** from `${U}/OPT/` — the ~20 INP files that configure each BPE step -2. The **`USER.CPU`** file from `${U}/CPU/` — the CPU slot configuration -3. The **`RUNBPE.INP`** from `${U}/PAN/` — the pre-configured BPE panel file - -These files become the ground-truth reference for the templates. Your production settings are preserved exactly — the orchestrator generates files that are functionally identical to what you would configure interactively. +*Rewritten 2026-08-03. The original version of this section asked for three files, +one of them at a path that does not exist and one that should not be copied +between machines at all. What is actually needed is now known precisely.* + +**The one thing still blocking a production run on the R740:** + +1. **`PAGENET_DLY.PCF`** — the daily Process Control File that drove the training + week. It exists **only on the T420**. It is the sequence of BPE steps for a + PAGENET day, and it must be copied rather than rebuilt: reconstructing it by + trimming the stock `RNX2SNX.PCF` leaves a step waiting on another step that no + longer exists, and the BPE then waits forever rather than failing. A rebuilt + file would also not be the one that has actually been proven to work. + +2. **`${U}/OPT/PGN_WK/`** — the weekly-combination panel directory, if it exists + on your machine. Expect our tooling to **reject it on the first attempt**: it + is known to contain Windows-style `\` path separators (literal characters on + Linux), a reference to a step that was removed, and session dates hardcoded + from the instructor's demo week. Being rejected is the tool doing its job. + The offending lines get remapped once, and then the corrected version is the + one everybody uses. + +**What we specifically do *not* want copied:** + +- **`USER.CPU`.** The original asked for this from `${U}/CPU/` — a directory that + does not exist (the file lives in `${U}/PAN/`). More importantly, it records + **how many CPU cores to use**, which is a property of the machine, not of the + processing. Copying it between machines is how the R740 came to be running the + laptop's setting of 2 — using two of its twelve cores on a step that already + takes forty minutes. It is now generated automatically from whatever hardware + the job runs on. + +- **Stock Bernese panels.** `${U}/OPT` and `${U}/PCF` on the R740 are currently + byte-identical to what Bernese 5.4 ships. We only want the files PHIVOLCS has + actually changed; the vendor's own files we already have. + +Everything supplied is version-controlled at `config/bernese/gpsuser/` and applied +by a single command, so the environment can be rebuilt from scratch if the server +is ever reconfigured — which matters, because the MIS team does reconfigure it. +Your production settings are preserved exactly; nothing is silently reinterpreted. --- diff --git a/docs/gnss_automation_roadmap.md b/docs/gnss_automation_roadmap.md index 0e2997d..58d0259 100644 --- a/docs/gnss_automation_roadmap.md +++ b/docs/gnss_automation_roadmap.md @@ -1,7 +1,40 @@ -# GNSS Data Processing Automation Roadmap +# GNSS Automation — Design Backlog ## MOVE Faults Project - PHIVOLCS -> **Context**: This document outlines automation opportunities for the existing Bernese GNSS 5.2 workflow, preserving the core processing engine while modernizing data ingestion, quality control, and metadata management. +**Written:** 2026-01-26 **Reclassified:** 2026-08-03 — *this is no longer the +project roadmap.* + +> ### Read this first +> +> **The live roadmap is [`project_documentation/roadmap.md`](project_documentation/roadmap.md), +> and current status is in [`project_documentation/deliverables_tracker.md`](project_documentation/deliverables_tracker.md).** +> Both are maintained. This document is not. +> +> This file and `roadmap.md` were written on the same day from the same material. +> `roadmap.md` is the restructured version that survived — reorganised around +> dependency tiers, kept current, and referenced throughout the project. This one +> was left behind and drifted for six months without anyone noticing, which is +> what prompted the reclassification rather than a rewrite. +> +> **It is kept because its contents are still worth having.** The detailed feature +> designs here — OCR for legacy log sheets, ML outlier detection, the QR equipment +> inventory, the notification matrix — have no equivalent anywhere else in the +> project. They are ideas with enough thought behind them to be picked up later. +> +> **What is definitely stale:** +> - **The phasing in §12.1** (Phase 1 = "Months 1–3") counted from January 2026. +> Those months have passed and the work happened in a different order, driven by +> what turned out to be urgent rather than by this plan. +> - **The effort estimates** were never checked against what things actually took. +> - **Anything implying nothing exists yet.** Several items here are built: +> `vadase-rt-monitor`, `drive-archaeologist`, the field-ops PWA, the +> `velocity-reviewer`, and the Bernese orchestrator — which now runs on a +> dedicated R740 with Bernese 5.4 verified to 0.0000 mm. +> +> Treat everything below as **a menu of designs, not a schedule.** Before starting +> anything from it, check the tracker for what already exists. + +> **Original context**: This document outlines automation opportunities for the existing Bernese GNSS 5.2 workflow, preserving the core processing engine while modernizing data ingestion, quality control, and metadata management. (The processing engine is now Bernese **5.4**.) --- diff --git a/docs/gps3-sessions/SESSION_LOG_20260729_storage.md b/docs/gps3-sessions/SESSION_LOG_20260729_storage.md index bb55fe0..83154ab 100644 --- a/docs/gps3-sessions/SESSION_LOG_20260729_storage.md +++ b/docs/gps3-sessions/SESSION_LOG_20260729_storage.md @@ -1,9 +1,34 @@ -# gps3 Session Log — 2026-07-29: Storage Provisioning +# gps3 Session Log — 2026-07-29 to 08-03 + +**Session:** `dell-gps` (Claude Code running on gps3, the Dell R740) +**Started as:** carve the unallocated PERC volume into LVs, migrate GPSDATA, +re-verify Bernese. **Grew into:** drive-health monitoring, an agency-side git +mirror, and the Bernese orchestrator's first contact with real data. -**Session:** `dell-gps` (Claude Code running on gps3) -**Purpose:** carve the unallocated PERC volume into LVs, migrate GPSDATA, re-verify Bernese. **Prior:** `~/HANDOVER.md` (updated 2026-07-29 by the T420 session) -**Outcome:** ✅ complete — new mounts in service, BPE numerical parity preserved. + +## Where to start reading + +This log is long because it is the succession record, not a changelog. If you +are new, read these three and skip the rest until you need it: + +| If you want | Read | +|---|---| +| What state the machine is in now | **§15** — end-of-session state and the one open blocker | +| Why the RAID is considered safe | **§13.2** — surface scanning, settled by measurement | +| Why the orchestrator could not read real data | **§14.3–14.5** | +| The mistake this project keeps making | **§15.5** — five instances, one shape | + +Sections are chronological and append-only. Where a later finding overturns an +earlier one, the earlier section carries a correction pointing forward rather +than being rewritten — the wrong turns are part of the record. + +**Outcome (2026-08-03):** storage in service with BPE numerical parity +preserved; all 16 RAID members monitored and confirmed surface-scanned; git +mirrored onto agency hardware; the Bernese orchestrator validating all seven +PAGENET sessions against the real DATAPOOL. Outstanding: the legacy archive is +still single-copy on failing media, and `PAGENET_DLY.PCF` is still only on the +T420. --- @@ -295,7 +320,7 @@ alerts begin mailing if an MTA is ever installed. --- -## 10. State at end of session +## 10. State at end of session — 2026-07-29 **Applied and verified:** @@ -473,3 +498,663 @@ found *before* the rebuild, which is what a surface scan is for. read is enabled and note its period. If it is disabled, either enable it, or append staggered long tests to the `-s` expressions: `-s (S/../.././NN|L/../../6/NN)`. + +> **RESOLVED 2026-08-03 — see §13.2. Do not act on the paragraph above.** +> The premise was wrong in a way worth noticing: it assumed surface scanning +> could only come from the controller, so an unreachable iDRAC meant an +> unanswerable question. It came instead from **BMS, the drive firmware's own +> Background Media Scan**, which runs below the controller and is fully visible +> from the OS. Full-surface reads are happening roughly daily on all 16 +> members. Long self-tests were considered and **rejected** — they would +> duplicate existing coverage. Whether PERC patrol read is enabled remains +> unknown and no longer matters. + +--- + +## 13. Mirror, surface-scan resolution, and archive prep — 2026-07-30 to 08-03 + +Covers the tail of the 07-30 session, a four-day unattended gap, and the +08-03 resumption. §1–12 stand as written except where noted above. + +### 13.1 Git mirror onto agency hardware + +The repository's only durable copy lived on a personal GitHub account — +identified in `COORDINATION.md` as the project's top continuity risk, since +nobody at PHIVOLCS can grant or recover access to it. There is now a bare +mirror on agency-owned storage: + +``` +/srv/gnss-archive/git/movefaults.git bare, ~14 MB, 50 branches at creation +/srv/gnss-archive/git/mirror-update.sh refresh script +cron: 37 22 * * * nightly +``` + +The design goal is **preservation, not synchronisation**, and the difference is +the whole point: + +| Setting | Value | Why | +|---|---|---| +| `--prune` on fetch | **omitted, deliberately** | A mirror that prunes faithfully reproduces upstream deletions. A force-push or branch deletion on GitHub would erase the same history here — the mirror would dutifully destroy the thing it exists to protect. | +| `core.logAllRefUpdates` | `true` | Bare repos default this **off**. With it on, every ref movement is recorded in the reflog, so a clobbered ref is still recoverable locally. | +| `gc.pruneExpire` | `never` | Unreachable objects are never garbage-collected. Anything that ever arrived stays. | +| `git fsck` | every run | Fixity check. Catches silent corruption on the array itself. | + +The combined effect is an **append-only** copy: upstream can add history to it, +but cannot take history away. The cost is that genuinely dead branches +accumulate, which is a trivial price. + +Two smaller guards, both from bugs this project has already paid for: + +- **Mount guard.** The script refuses to run unless `/srv/gnss-archive` is a + real mountpoint. Without it, an unmounted array means the mirror silently + writes into the empty directory on the root filesystem — appearing to work + while backing up nothing to the wrong disk. +- **`BatchMode=yes`, `GIT_TERMINAL_PROMPT=0`.** Under cron there is no tty; a + credential prompt would hang the job indefinitely instead of failing. Fail + fast and loudly beats hang quietly. + +The script is committed as `scripts/gnss_mirror_update.sh`. **The deployed copy +at `/srv/gnss-archive/git/` is the one cron runs** — edits to the repo copy do +not take effect until copied across. + +**Verified 2026-08-03 after four unattended days:** fired all four nights, +`fsck clean` every run, branches 50 → 54. It works without supervision, which +was the requirement. + +### 13.2 Surface scanning: the §12.4 gap closed, from an unexpected direction + +§12.4 left a genuine hole. smartd schedules only short self-tests; long tests +were omitted on the assumption that PERC patrol read handles surface scanning; +that assumption was never checked, and appeared uncheckable — no vendor CLI +exists in the Ubuntu repos and iDRAC has no IP (§13.4). + +**The question was answerable after all, because the assumption about where +scanning comes from was too narrow.** SAS drives run their own **Background +Media Scan (BMS)** in firmware, below the RAID controller, and both the scan +log and the lifetime byte counters are readable through the megaraid +pass-through. Measured 2026-08-03 via `/home/gps3/patrol_check.sh`: + +``` +Status: waiting until BMS interval timer expires +Accumulated power on time, hours:minutes 10142:56 +Number of background scans performed: 424 + +read: 0 228 228 228 3929 1025339.394 0 + GB processed ^ ^ uncorrected +``` + +| Measure | Value | +|---|---| +| Lifetime read, per drive | ~1,025,300 GB ≈ **1.03 PB** | +| Total uncorrected read errors | **0** on all 16 | +| Background scans performed | **424** | +| Power-on hours | 10,142 (≈423 days) | +| Implied scan interval | **≈ one full pass per 24 h** | + +**The cross-check that makes this a conclusion rather than a guess.** Scan +count and bytes-read are separate firmware counters kept for unrelated +purposes. 424 scans × 2.4 TB = 1,017.6 TB, against a measured 1,025.3 TB — +agreement within 0.8%. Two independent counters converge on the same fact, so +the conclusion does not rest on trusting either number's label. `patrol_check.sh` +now computes this ratio directly and prints it as the `SWEEPS` column. + +**Direct evidence the mechanism has already earned its keep.** Across all 16 +members BMS has found and repaired **~112 physical sectors** (895 logical +entries), every one logged `[1,18,7] Recovered via rewrite in-place` — +RECOVERED ERROR, data retrieved via ECC and retries, then rewritten. **Zero +reassignments, zero failures, zero uncorrected errors array-wide.** These are +exactly the latent defects that destroy a 16-wide RAID 5 rebuild, caught during +routine scanning while every member was healthy and full parity was available. +The ~3.1% per-rebuild URE risk from §2 of `COORDINATION.md` is being actively +worked down, not merely assumed away. + +**Read this column in physical sectors, not log entries.** These are 512e +drives — one 4096-byte physical sector reports as **eight** consecutive +512-byte logical entries. The distribution looked alarming at first: + +| Member | Logical entries | Physical sectors | +|---|---|---| +| **6** | 671 | ~84 | +| 1 | 96 | 12 | +| 8, 9 | 40 each | 5 each | +| 15 | 24 | 3 | +| 7 | 16 | 2 | +| 5 | 8 | 1 | +| 0, 2–4, 10–14 | 0 | 0 | + +Member 6 at seven times its nearest peer was investigated as a probable early +failure (`scripts/sudo/inspect_member6.sh`). **It is healthy.** All 671 entries +are rewrite-in-place; nothing was reassigned, so no spare sectors were consumed +and the grown defect list is not growing. The defects span power-on hours 121 +to 10,127 — the drive's *entire* life at a steady ~1 physical sector every five +days — rather than clustering recently, which is what degradation looks like. +And 671/8 ≈ 84 matches that drive's own corrected-read-error counter of exactly +84, an independent confirmation of the unit factor. **No replacement needed.** + +The instructive part is that the ranking is misleading in both directions. +Member 0 — zero BMS entries, the apparent control — carries **228** corrected +read errors and **3,929** correction-algorithm invocations, against member 6's +84 and 85. Member 0 works considerably harder to read its own data; it simply +never needed a rewrite. **Ranking drives by defect count alone selects the +wrong suspect.** What actually indicates a failing drive is a `reassign_status` +other than rewrite-in-place, defects concentrated in recent power-on hours, or +any non-zero uncorrected count. Member 6 has none of the three. + +**Decision: do not add SMART long tests.** The `-s (S/../.././NN|L/../../6/NN)` +change contemplated in §12.4 is **rejected** — it would duplicate a daily +full-surface scan and add contention for no additional coverage. The smartd +config in §9 stands unchanged. + +**What remains genuinely unknown:** whether PERC patrol read is enabled. BMS +says nothing about it either way. It no longer blocks anything, since both +defend the same failure mode and one of them is confirmed running. + +### 13.3 A fourth inverted-value bug, in the tool built to find the third + +Worth recording because of where it happened. The first run of +`patrol_check.sh` reported **0.00 TB read and ~1,025,339 uncorrected errors per +drive** — a fleet of unscanned, catastrophically failing disks. Both figures +were wrong, and they were the same two numbers with their columns transposed: + +``` +read: 0 228 228 228 3929 1025339.394 0 + ^ $(NF-1) ^ $(NF) + GB processed uncorrected errors +``` + +The script read `$(NF)` as gigabytes and `$(NF-1)` as uncorrected errors. + +Three things make this worth more than a one-line fix: + +1. **The output was individually plausible.** A near-zero read total is what an + unscanned array looks like. A huge error count is what a dying drive looks + like. Nothing was obviously malformed — only the *magnitude* of the error + count (a million errors on a drive still answering queries) gave it away. +2. **It reached the exactly-opposite conclusion.** Not a degraded or partial + answer: a confident, precisely inverted one, on the specific question the + tool existed to settle. +3. **This script was written to close out the third instance of this bug + class** (the SIGPIPE/pipefail bug in §12.1), and its header comment warns + about that bug — while committing a fresh variant of the same family + fourteen lines below the warning. + +The family: **a value taken from the wrong place, then trusted because it +looked reasonable.** SIGPIPE 141 read as failure; `lsof +D` exit 1 read as +"files open"; a trailing `[ cond ] && echo` read as the script's verdict; and +now `$(NF)` read as gigabytes. Knowing the pattern did not prevent it. + +What did catch it: **going back to the raw tool output instead of iterating on +the parsed summary.** The fix is now pinned in the script by a comment carrying +the literal column header, so the next reader can check the field positions +without re-deriving them. + +**A second misreading, same session, same shape.** With the parse fixed, the +recovered-sector column was read in the wrong *units* — raw 512-byte log +entries rather than 4K physical sectors — making member 6 look seven times +worse than its peers and very nearly justifying a proactive drive replacement +(§13.2). Not a code bug this time: the script reported exactly what it +measured. The error was in interpretation, and it produced the same outcome as +the parse bug — a confident conclusion, drawn from a real number, pointing the +wrong way. + +What broke both: **going back to the raw per-drive output instead of iterating +on the summary.** Worth generalising, since it is now the operating lesson of +this whole section — *a derived number is only as good as the last time someone +checked it against the thing it was derived from.* Both fixes are pinned in the +script by comments carrying the literal evidence (the column header; the +671 ÷ 8 = 84 cross-check), so the next reader can verify without re-deriving. + +`patrol_check.sh` was corrected to report scan count, recovered sectors in +physical units, and the sweeps ratio, and is now in the repo at +`scripts/patrol_check.sh`. Root-requiring run-scripts live in `scripts/sudo/` +(`verify_patrol_check.sh`, `inspect_member6.sh`), with their logs gitignored +and the scripts themselves committed as the record of what was run. + +### 13.4 iDRAC is not reachable + +``` +sudo ipmitool lan print 1 -> IP Address 0.0.0.0, IP Address Source: DHCP + MAC b0:7b:25:fe:2c:38 +``` + +`/dev/ipmi0` exists and the drivers are loaded, so the BMC is alive — it simply +has no address. **There is no out-of-band management on this machine today:** no +remote console, no remote power cycle, no hardware event log access, no +controller configuration. If gps3 fails to boot, it needs someone physically in +front of it. + +`sudo ipmitool delloem lan get` will report whether the BMC is on the dedicated +port or shares an onboard NIC, which determines what needs cabling. + +**Before it is put on the network, change the password.** Dell ships iDRAC with +factory credentials `root`/`calvin`, and an iDRAC is a full remote console with +power control — reachable regardless of the OS state. + +### 13.5 Branching policy adopted + +Merged to `main` 2026-07-30 (PR #58) and now in `CLAUDE.md`: all substantive +work reaches `main` through a PR; branches live at most one week; `git pull +--rebase` before every push; never redirect a gated git/gh operation to +`/dev/null`; verify after every merge and retarget. + +Rule 2's one-week limit came from `docs/bernese-training-notes` drifting 27 days +until neither it nor `main` held the whole project, taking a full session (PR +#57) to reconcile. The diagnosis worth keeping: the failure was not the branch, +it was the branch **outliving its purpose and quietly becoming a second trunk**. + +Commits `6c7709c` and `23d4b29` predate the policy, went directly to `main`, and +still carry AI-attribution trailers the policy now forbids. Left alone +deliberately — rewriting `main` to tidy history is a worse act than the +inconsistency it would fix. + +### 13.6 Archive: receiving side ready, still empty + +`/srv/gnss-archive/legacy` is prepared and **verified empty (0 entries)** as of +2026-08-03. The ~157 GB legacy archive still exists in exactly one place: a +personal external drive with a pending sector. **This remains the single +largest data-loss risk in the project.** Blocked on the push from the T420's +DOSTB mount. + +`/srv/gnss-archive/verify_archive.sh` is staged for afterwards, in two modes: + +- **`census`** — counts files, symlinks, directories and bytes separately, to + be compared against the same census on the source. Separate counts because + this archive once lost every symlink to a FAT32 hop, silently: a file count + alone would not have noticed, since the symlinks were still present as + regular files. +- **`manifest`** — sha256 over the destination, written to + `/srv/gnss-archive/manifests/` and gzipped for committing to git. + +**Why a census and not `rsync --stats`:** rsync exits 0 having skipped files it +could not read, and exits 23 on a run that copied 99.99% successfully. Neither +number describes what actually landed. Hashing is destination-only on purpose — +the source is failing media being rescued, rsync already verifies each file in +flight, and the manifest's real job is detecting silent corruption on the array +years from now, which nothing currently defends against. + +**Commit the manifest `.gz` to git.** Fingerprints stored only beside the data +cannot prove anything if that disk is what went wrong. + +Note the third exit-status bug of the set was found here, in this script, by +running it against the empty destination before handover: a trailing +`[ "$other" -gt 0 ] && echo …` made a *clean* census exit 1. + +### 13.7 Four unattended days, and the gh token + +The 07-30 session ended with the laptop leaving the network. Nothing was in +flight. Verified on return 2026-08-03: mirror cron fired all four nights, +`smartd` still active, no alerts, uptime 11 days. + +**Lesson recorded:** that session ran outside `tmux`, so disconnecting ended it. +Nothing was lost because nothing was running — but the archive transfer will be +hours of rsync from failing media, and must run inside `tmux` on both ends. +Start `tmux` *first*, then the work inside it; a running process cannot be +moved in afterwards. + +`gh`'s stored token had gone invalid over the gap (`HTTP 401: Bad credentials`). +Git itself was unaffected — it uses SSH — so the failure surfaced only when a +PR wrapper was called. Re-authenticated via device flow. Note that `gh` stores +the new token **in plain text** in `~/.config/gh/hosts.yml`; that is normal `gh` +behaviour, not a misconfiguration, but it is a second static credential on this +box alongside `R740_PASS` in `scripts/deploy_r740.secrets` (still `gps3`, the +same as the username, on a LAN-reachable host with sudo — should be changed). + +### 13.8 Infrastructure state + +| Item | State | +|---|---| +| smartd | Active, 16 members, staggered short tests, alert path proven | +| Surface scanning | **Confirmed** — BMS, ~daily full pass, SWEEPS 1.01x on all 16 | +| Drive health | 0 uncorrected errors array-wide; ~112 sectors repaired in place, none reassigned; member 6 investigated and cleared | +| Git mirror | Nightly, 4/4 runs clean unattended, append-only by design | +| `/srv/gnss-archive/legacy` | **Empty** — transfer not started | +| iDRAC | No IP; no out-of-band management exists | +| PERC patrol read | Unknown; no longer blocking | +| Kernel | 6.8.0-136 installed, **6.8.0-111 running** — reboot pending | +| `~/GPSDATA.old-20260729` | 4.5 GB retained; this *is* the migration rollback | + +The Bernese work that followed on the same day is §14; the consolidated +end-of-session state and handover is §15. + +--- + +## 14. Bernese orchestrator deployment — 2026-08-03 + +Working against §5 of `docs/project_documentation/bernese_orchestrator_r740_readiness.md`. + +### 14.1 Where the deployment actually stands + +**Step 1 (install + verify) is done** — `BERN54` present, EXAMPLE campaign +verified at 0.0000 mm on 07-29. Everything needed for the acceptance test is +already on the box, which was not obvious before checking: + +| Asset | State | +|---|---| +| PAGENET RINEX | **677 files, DOY 081–090** in `$D/PGN` — wider than the 084–086 training week | +| `PGN.*` reference set | STA, CRD, ABB, CLU, BLQ, ATL, PLD, VEL — all in `REF54/` | +| Orchestrator P0 tasks A–E | All have code; **128 tests pass** | +| DATAPOOL migration (07-29) | **Verified complete** — 0 diff lines old vs new, 4.1 G both | +| `GPSDATA` volume | Own 4 TB LV, 4.5 G used — DL-012 disk pressure is **not** a near-term constraint | + +**Correction (made later the same day, §14.4).** An earlier reading of this +recorded that `PLG2` — the station that hard-aborted DOY 086 on the T420 — was +"absent from this DATAPOOL entirely (0 files)". That was wrong, and wrong in an +instructive way: `ls | grep -i plg2` at the top level finds nothing because the +files are **hand-quarantined in a hidden subdirectory**, +`.excluded_plg2/plg20860.26o.gz` and `plg20880.26o.gz`. They came across in the +migration intact. PLG2 is still **missing from `PGN.STA`**, so the underlying +defect is unfixed; it is merely hidden behind a manual workaround applied +during the training week. + +The reference files also disagree with each other — `PGN.STA` 74 records, +`PGN.CRD` 72, `PGN.ABB` 71, against 71–72 stations per session in the RINEX. +A `.STA` carrying more stations than any one day's data is normal and benign; +validation against all seven sessions now passes clean (§14.4). + +### 14.2 maxjobs was 2 — the R740 was using 2 of its 12 cores + +`USER.CPU` carried the **T420's** `maxjobs 2` across with the config. The 502 +GPSCLU_P bottleneck (readiness §2.4, ~40 min of every ~2 h run) was therefore +being served by a box configured as if it were the laptop. + +Corrected via the repo's own `cpu_config.compute_maxjobs()` rather than by hand, +so the change is the one the orchestrator will make in production: + +``` +physical cores=12 ram=62.0G reserve=1 -> maxjobs=11 +"localhost" "…" "FAST" "11" "0" "0" (was "2") +``` + +Backup at `PAN/USER.CPU.bak-`. + +**The readiness doc's core count was wrong by 2×** — it assumed 24 physical from +the gaps memory. `lscpu`: Xeon Silver 4214R, 1 socket, **12 physical**, 2 +threads/core = 24 logical. Since maxjobs tracks physical cores (sub-solves are +FPU-bound and gain nothing from hyperthreads sharing an FPU), setting 24 would +have oversubscribed by 2× and plausibly run *slower* than a correct 12. Doc +corrected in place. Also settled: the CPU carries AVX-512 (Cascade Lake), so the +x86-64 ISA `objcopy` patch that section flagged as uncertain is **not needed**. + +`V_CLUFIN` clustering (P2-K) is still untuned and remains the other half of the +502 fix. + +### 14.3 The validator cannot see the real DATAPOOL — new P0 blocker + +Pointing `validate_rinex_headers()` at the actual gps3 DATAPOOL: + +``` +ERROR No RINEX observation files found in /home/gps3/GPSDATA/DATAPOOL/PGN + for session 2026/0860 — refusing to pass validation vacuously +``` + +`_is_rinex_obs()` matches on `path.suffix` against `.rnx`, `.obs`, `.rxo`, +`.o`. **Every file here is gzipped**, so the suffix is `.gz`. Decompressing +would not save it: PAGENET files are Hatanaka `.26d` and the IGS fiducials +`.crx`, neither of which is accepted. The real names are `PZAM0860.26d.gz`, +`pbay0860.26o.gz`, `CUSV00THA_R_20260860000_01D_30S_MO.crx.gz`. + +**Why this is worse than an ordinary bug.** It surfaced loudly only because the +call passed `require_stations=True`. Under the **default** `require_stations=False` +the function returns a **passing** report — approving every session while having +examined nothing, with the first symptom being RXOBV3 hard-aborting mid-BPE. +That is the "vacuous pass" the docstring itself warns about, reached through a +door nobody anticipated. + +And **the 128 tests pass**, because the fixtures use uncompressed `.YYo`/`.rnx` +names. The gap is invisible to the suite and appears only against real data — +which is readiness §6's thesis, demonstrated on the first contact with the +production DATAPOOL. + +Filed as task **C2**. Fix: strip `.gz`/`.Z` before matching, accept `.d` and +`.crx`, decompress (or header-read) via `gzip` + `CRX2RNX`, and add fixtures in +the real naming scheme so the suite can catch this class. + +This is also a fifth instance of the session's running theme — see §13.3. A +check that reports success without having inspected anything is the same defect +as an exit status that reports success without having run anything. + +### 14.4 C2 fixed — the validator now sees the real DATAPOOL + +**Result: all seven PAGENET sessions (DOY 084–090) validate clean**, from a +starting point of zero files visible. 179 tests pass, up from 128. + +**The fix was much smaller than expected, because of one property of CRINEX.** +A Hatanaka file stores the original RINEX header **verbatim** after two +`CRINEX VERS`/`CRINEX PROG` lines; only the observation records *below* +`END OF HEADER` are compacted. Since this validator reads nothing past the +header, `crx2rnx` never has to run. No RNXCMP build, no `hatanaka` package, no +Hatanaka decoding anywhere in validation — **decompression alone is enough.** +(RNXCMP is still needed for actual processing; canonical source is GSI's RNXCMP +page, currently 4.1.0, plain C with no dependencies.) + +**Compression had to handle two formats, not one.** IGS convention is `.Z` +(UNIX compress / LZW), not `.gz`. Python's `gzip` module cannot read it — +`BadGzipFile: Not a gzipped file (b'\x1f\x9d')`, LZW magic `1f 9d` against +gzip's `1f 8b`. On this box: 3,010 `.gz` against 20 `.Z`, including four real +Hatanaka-plus-LZW files at `GRCC/RINEX/GFCN0100.23D.Z`. Resolution: Python +`gzip` for `.gz` (keeping the 3,010-file path subprocess-free), GNU `gzip -dc` +for `.Z`. No new dependency. + +Extension stripping loops right-to-left rather than using one regex, because +every real name stacks two extensions and both orders occur: `PZAM0860.26d.gz`, +`GFCN0100.23D.Z`, `CUSV..._MO.crx.gz`, `..._MO.rnx.gz`. + +**Two further defects surfaced only once the validator could see data at all.** +Both would have blocked the acceptance test, and neither was visible before: + +1. **Descriptive marker names shadowed the station code.** PAGENET CORS write + `MARKER NAME = "BOGO CITY"` with the code in `MARKER NUMBER = "PBOG"`; IGS + fiducials do the reverse (`MARKER NAME = "CUSV"`, `MARKER NUMBER` = a 9-char + DOMES). Taking `MARKER NAME[:4]` yielded `BOGO`, absent from `PGN.STA`, so + **9 of 72 real stations were reported missing on data that processes + correctly**. Two naming conventions in one campaign — readiness §2.6 again. + `_resolve_station_code()` now prefers a bare 4-char `MARKER NUMBER`, then a + bare 4-char `MARKER NAME`, then the filename, and logs disagreements. + *A validator that fails on good data gets switched off, which costs more + than the check was ever worth.* + +2. **`rglob` descended into the hidden quarantine directory.** It picked up + `.excluded_plg2/`, reporting PLG2 missing from `PGN.STA` on exactly DOY 086 + and 088 — which, pleasingly, reproduces readiness §2.2's empirical finding + ("present only DOY 086 + 088") from an entirely independent direction. But + RNX_COP globs the source directory *without* recursing, so those files will + never be staged. **The validator must model what will actually be + processed**; flagging files that cannot reach the run is the mirror image of + the vacuous pass, and just as effective at getting the check ignored. + Dot-directories are now skipped, with the reasoning recorded in the code. + +**PLG2 remains genuinely missing from `PGN.STA`.** The quarantine is a manual +workaround from the training week, not a fix, and task A is meant to replace it +with automatic per-session detection and quarantine. Skipping dot-directories +suppresses the *symptom* in validation; it does not resolve the defect. + +Test coverage now spans the production filename space — the reason the previous +128 could not see any of this. Added: parametrised recognition across all real +encodings and their negative cases (nav/met/product files), round-trip reads for +plain/`.gz`/`.Z`, a from-scratch LZW encoder so `.Z` is testable with no +external compressor (nothing on stock Ubuntu can *write* `.Z`), an early-exit +test asserting that SIGPIPE from the cut-off `gzip` is not treated as failure, +and an integration test against the real gps3 DATAPOOL that skips off-host. + +### 14.5 Provisioning `$U` — mechanism built, one asset still missing + +§5 step 2 says "provision `$U` from repo gold-standard PCFs/panels/scripts". +**The gold standard did not exist.** Checking first was worth it: + +- `$U/OPT`, `$U/PCF`, `$U/SCRIPT`, `$U/PAN` on gps3 are **byte-identical to the + `$C/USER` template** shipped with Bernese 5.4 — zero files differing. Nothing + PHIVOLCS-specific had ever been deployed here. +- The repo held only `scripts/pagenet_pcs.pl` and one Jinja template. P1-H's + "gold-standard panels versioned in repo" was aspirational. + +**Built the mechanism** (`config/bernese/gpsuser/` + `scripts/provision_gpsuser.py`), +because it is the thing readiness §4 actually requires: after a MIS reset, a +working environment must be recoverable by re-running provisioning rather than +by re-debugging panels by hand. + +Three file classes, handled deliberately differently: + +| Class | Treatment | Why | +|---|---|---| +| `OPT/**/*.INP` | Separator-sanitized; `ADDNEQ2.INP` MAXPAR sized from station count | Windows `\` are literal chars on Linux (gap #8, readiness §2.5) | +| `SCRIPT/*` | Copied **verbatim** | A backslash in Perl is an escape, not a path separator — converting corrupts the driver | +| `PCF/*.PCF` | Checked for dangling `WAIT`, refused if any | A WAIT on an undefined PID makes the BPE block **forever**, silently | + +`PAN/USER.CPU` is **generated, never versioned.** `maxjobs` must track the +host's physical cores, so a committed copy would carry one machine's core count +onto another — which is precisely the bug found earlier today (§14.2). The +provisioner detects cores and RAM itself and independently arrived at +`maxjobs=11`, matching the hand-set value. + +Dry-run by default; `--apply` to write. Strict: a panel with an unresolvable +hazard aborts the whole run *before* anything is written, so `$U` is never left +half-updated. + +**Applied.** `pagenet_pcs.pl` is now at `$U/SCRIPT/`, byte-identical to the gold +copy, and a second run reports no changes. + +**Still blocked: `PAGENET_DLY.PCF`.** It exists only on the T420, where it drove +the full training week. **It must be captured, not re-derived.** It is described +as RNX2SNX modules 1–14 (PID 001→514). In stock `RNX2SNX.PCF`, `599 DUMMY` waits +on `512 514 522`, so naively dropping the R2S_RED branch (521/522) would leave +599 waiting on a PID that never runs. + +> **Corrected 2026-08-04, having now seen the real file (PR #65).** That was a +> prediction about what a careless truncation *would* do, and the actual +> `PAGENET_DLY.PCF` does not have the problem: its `599 DUMMY` waits on +> `512 514`, and 521/522 are simply absent. `find_dangling_waits()` reports +> **zero**. Whoever produced it performed the reduction properly rather than +> cutting the file short. +> +> The advice to capture rather than re-derive still stands, but the honest +> reason is weaker than the one originally given: not "a truncation leaves a +> dangling WAIT" — this one demonstrably does not — but that the captured file +> is the one actually validated during the training week, and the `9xx` +> save/cleanup tail involves choices a reconstruction would have to guess at. +> A stated hazard that turns out not to apply is worth less than it appears, +> and worth correcting at the point it was claimed. + +To hand it over, from the T420: + +```bash +cp "$U/PCF/PAGENET_DLY.PCF" /config/bernese/gpsuser/PCF/ +cp -r "$U/OPT/PGN_WK" /config/bernese/gpsuser/OPT/ # if present +``` + +Expect the provisioner to **reject `PGN_WK/ADDNEQ2.INP` on first attempt** — +readiness §2.5 records it carrying Windows separators, a dangling `WAIT=522`, +and hardcoded sessions (`20261030/40/50`, the instructor's demo week). That +rejection is the tool working, not a malfunction: remap the hardcoded literals, +then re-run. + +### 14.6 Still to do for BRN-001 + +1. **Capture `PAGENET_DLY.PCF` from the T420** into + `config/bernese/gpsuser/PCF/` — the only thing between here and an acceptance + test. +2. **Add PLG2 to `PGN.STA`** (or implement task A's automatic quarantine) and + retire `.excluded_plg2/`. +3. **Tune `V_CLUFIN`** (P2-K) — empirical, needs a real run to measure. +4. **Acceptance test**: one PAGENET session end-to-end on gps3, then the week. + It must clear the station/MAXPAR/panel problems *automatically*, not by hand. + +--- + +## 15. End of session — 2026-08-03 + +### 15.1 What changed on the machine + +Everything below is applied and verified, not merely written down: + +| Change | Verification | +|---|---| +| `USER.CPU` maxjobs **2 → 11** | Set via `cpu_config.compute_maxjobs()`; backup at `PAN/USER.CPU.bak-*` | +| `pagenet_pcs.pl` deployed to `$U/SCRIPT/` | `cmp` byte-identical to gold copy; re-run reports no change | +| `patrol_check.sh` corrected, moved into repo | Re-run: SWEEPS 1.01x, UNCORR 0 on all 16 | +| `scripts/sudo/` convention established | Two scripts used in anger this session | +| `.gitignore` — `scripts/sudo/logs/` | Logs excluded, scripts committed | + +**Nothing was changed on the array, the mirror, or the archive.** The storage +side was read-only this session. + +### 15.2 Questions that were open this morning and are now closed + +- **Does anything scan the RAID surface?** Yes — BMS, ~daily, all 16 members, + confirmed by two independent counters. Long self-tests explicitly rejected. + (§13.2) +- **Is member 6 failing?** No. The 671 figure was 512-byte logical entries; + ~84 physical sectors, flat rate across the drive's whole life, nothing + reassigned. (§13.2) +- **Did the 07-29 GPSDATA migration lose anything?** No — 0 diff lines between + old and new DATAPOOL. (§14.1) +- **How many cores does the R740 actually have?** 12 physical / 24 logical. The + readiness doc's 24 was the logical count. (§14.2) +- **Is the objcopy ISA patch needed?** No — AVX-512 present. (§14.2) +- **Can the orchestrator read real GNSS data?** It could not; now it can. + All seven PAGENET sessions validate clean. (§14.4) +- **Does a repo gold standard for `$U` exist?** It did not; it does now. (§14.5) + +### 15.3 The one thing blocking progress + +**`PAGENET_DLY.PCF` exists only on the T420.** Until it is committed to +`config/bernese/gpsuser/PCF/`, no acceptance test can run on gps3 — the +orchestrator is otherwise ready, `$U` is provisioned, `maxjobs` is correct and +the validator works against real data. Two `cp` commands on the other machine +close it (§14.5). + +Do **not** re-derive it. §14.5 explains why, and the provisioner will now refuse +a truncation that leaves a dangling WAIT. + +### 15.4 Delivered + +> **Correction 2026-08-04.** The sentence below originally read "All work +> reached `main`". It had not, and still has not. Everything is on the branch in +> an **open** PR; `origin/main` was and is at `1d1082e` (PR #60). Rule 5 exists +> for exactly this — *verify `origin/main` actually advanced*, do not infer it +> from having pushed successfully. Caught by the T420 session, not by me, in a +> section I wrote to be the authoritative end-of-session state. + +All work is queued for `main` in [PR #64](https://github.com/alfieprojectsdev/movefaults/pull/64), **still open**, +branch `docs/gps3-session-20260803`, three commits: + +| Commit | Contents | +|---|---| +| `a82be79` | §13, tmux runbook, `patrol_check.sh`, `scripts/sudo/` | +| `ad2401c` | C2 — validator blind to real DATAPOOL; +51 tests | +| `3f51cd7` | `$U` gold standard and provisioner; +10 tests | + +Test count 128 → 189, `ruff check` clean on everything touched. + +Also written this session: `docs/gps3_tmux_claude_runbook.md`, a from-scratch +tmux guide for gps3 aimed at someone who has never used it — including the rule +that costs the most when broken (**start tmux first; a running process cannot be +moved into it afterwards**) and the fact that a reboot destroys every session +while cron and smartd carry on regardless. + +Three T420 PRs (#61, #62, #63) were open and unmerged at session end. They are +the other machine's to land. + +### 15.5 The running theme, for whoever reads this next + +This session found the same defect shape five times, and it is worth naming +because it will recur: + +> **A check that reports success without having inspected anything.** + +- `smartd`'s `DEVICESCAN` — a green service monitoring **zero** drives (§9) +- `slow_cmd | grep -q` under `pipefail` — success inverted to 141 by SIGPIPE (§12.1) +- `patrol_check.sh` — columns transposed, reporting an unscanned, dying array + that was neither (§13.3) +- The same script read in the wrong **units**, nearly condemning a healthy drive (§13.3) +- `validate_rinex_headers()` — a **passing** report having read no files at all (§14.3) + +In every case the output was individually plausible, and in four of the five a +passing test suite or a zero exit status actively concealed it. What broke each +one was the same move: **go back to the raw output of the underlying tool +instead of iterating on the summary.** The `smartctl` column header, the +measured exit codes, `671 ÷ 8 = 84`, the actual DATAPOOL filenames. + +The corollary for the test suite is sharper still. The 128 tests passed +throughout because their fixtures described a filename space that does not +exist in production. **A suite that never sees real data cannot fail on a +misreading of real data**, no matter how many assertions it contains. diff --git a/docs/gps3_tmux_claude_runbook.md b/docs/gps3_tmux_claude_runbook.md new file mode 100644 index 0000000..8910de9 --- /dev/null +++ b/docs/gps3_tmux_claude_runbook.md @@ -0,0 +1,201 @@ +# Running Claude Code on gps3 (Dell R740) under tmux + +**Audience:** anyone with an account on gps3 — including whoever inherits this +system. Assumes no prior tmux knowledge. + +**Why this document exists.** On 2026-07-30 a working session was run *outside* +tmux. When the laptop left the network, the SSH connection dropped and the +session ended with it. Nothing was lost that time because nothing was running — +but the archive transfer still ahead of us is hours of `rsync` from a failing +external drive, and losing that midway is expensive. tmux makes the difference +between "my laptop closed" and "the work stopped." + +--- + +## 1. The one rule + +**Start `tmux` FIRST, then start work inside it.** + +A process already running in a plain SSH session **cannot be moved into tmux +afterwards**. If you forget, you have to stop and restart the work. There is no +retrofit. + +```bash +ssh gps3 +tmux new -s claude # 1. create the session +claude # 2. THEN start Claude Code, inside it +``` + +--- + +## 2. Daily use + +| Action | Command | +|---|---| +| Start a new named session | `tmux new -s claude` | +| **Detach** (leave it running) | Press `Ctrl-b`, release, then press `d` | +| Reattach later | `tmux attach -t claude` | +| List sessions | `tmux ls` | +| Attach if exists, else create | `tmux new -A -s claude` | +| Kill a session you're done with | `tmux kill-session -t claude` | + +`Ctrl-b` is the tmux *prefix*: hold Ctrl, press `b`, **release both**, then +press the next key. It is not a chord — `Ctrl-b-d` pressed together does not +work. + +Detaching is safe and instant. Everything inside keeps running with no terminal +attached. You can close the laptop, lose wifi, or fly somewhere; the work +continues on the server. + +`tmux new -A -s claude` is the one to memorise if you only keep one: it attaches +to `claude` if it exists and creates it if it doesn't, so it is always correct. + +--- + +## 3. Resuming a Claude Code conversation + +Two independent things persist, and it helps to keep them straight: + +- **The tmux session** lives on gps3 and holds the *running process*. +- **The conversation** lives on Anthropic's servers and holds the *history*. + +So a conversation can be resumed even if tmux died: + +```bash +claude --resume # interactive picker of recent sessions +claude --resume # jump straight to a known one +claude --continue # most recent conversation in this directory +``` + +The session id appears in the Claude Code UI and in the transcript path under +`~/.claude/projects/`. Record it in your session log when you stop for the day — +it costs one line and saves hunting later. + +--- + +## 4. What survives what + +| Event | tmux session | Claude conversation | Remote control | +|---|---|---|---| +| SSH drops / wifi lost | **Survives** | **Survives** | **Survives** | +| Laptop closed or shut down | **Survives** | **Survives** | **Survives** | +| You detach with `Ctrl-b d` | **Survives** | **Survives** | **Survives** | +| You close the SSH window | **Survives** | **Survives** | **Survives** | +| You type `exit` in the shell | Ends | Survives (`--resume`) | Ends with the process | +| **gps3 reboots** | **Destroyed** | Survives (`--resume`) | **Destroyed** | + +**The remote-control column is measured, not assumed** (2026-08-03), in two +stages: + +1. **SSH window closed, same network.** `tmux list-clients` returned **no + attached clients**, yet the session still executed commands and answered + normally. Confirmed as continuation rather than restart by PID — the process + kept its original number (415338) with an uptime matching the tmux session's + creation time. +2. **Reconnected from a home network, entirely outside the PHIVOLCS LAN.** The + same session remained drivable with no VPN and no inbound port. + +The second result is the operationally important one. **A long job on gps3 can +be started at the office and supervised from home** — which is the difference +between the archive transfer being a single unattended run and being something +that has to be babysat on site. + +Why it works without any firewall change: gps3 makes an **outbound** connection. +Nothing listens for an inbound one, so this does not expose the machine to the +network it is reached from. The corollary is worth stating plainly: anyone with +access to the Claude account can drive this box, with whatever privileges the +session's user holds. + +Caveat on the first test: a second SSH session from another host was still open. +The tmux session had zero attached clients either way, so the result holds, but +it was not clean-room. The second test did not depend on it. + +**The reboot row is the one that catches people.** tmux keeps nothing on disk. +A kernel update or power event takes every session with it. Before a planned +reboot, stop long-running work deliberately rather than trusting tmux to carry +it through — it will not. + +Note also that **system services do not need tmux at all.** The nightly git +mirror (cron, 22:37) and `smartd` run independently of any login session and +keep working while nobody is connected. tmux is only for *interactive* work you +started yourself. + +--- + +## 5. Long transfers: the case tmux is actually for + +For anything measured in hours — the legacy archive transfer especially — use a +**dedicated session per job**, not the one you're chatting in: + +```bash +tmux new -s archive +rsync -av --info=progress2 / /srv/gnss-archive/legacy/ +# Ctrl-b d to detach; check back whenever +``` + +Then reattach with `tmux attach -t archive` to see progress. + +Run it inside tmux **on both ends** if the transfer is initiated from another +machine — a dropped SSH session kills the *client* side just as dead. + +**Do not trust `rsync`'s exit code as proof the copy is complete.** rsync exits +0 having skipped files it could not read, and exits 23 on a run that copied +99.99% successfully. Verify with an independent census afterwards: + +```bash +sudo -u gps3 /srv/gnss-archive/verify_archive.sh census +``` + +Compare files, symlinks, directories and bytes against the same census run on +the source. Symlinks are counted separately on purpose: this archive once lost +every symlink to a FAT32 intermediate hop, silently, and a file count alone +would not have noticed because the symlinks were still there as regular files. + +--- + +## 6. Running `sudo` commands during a Claude Code session + +Claude Code's shell has **no tty**, so it can never run `sudo` itself — the +password prompt has nowhere to appear. The established workflow on this box: + +1. Keep a **second terminal** SSH'd into gps3 (its own tmux session is fine). +2. Claude writes the command as a script under + `/home/gps3/repos/movefaults_clean/scripts/sudo/` and gives you the + **absolute path only**. +3. You run that path with `sudo` in your terminal. +4. The script `tee`s its output to `scripts/sudo/logs/`, so you watch it live + *and* Claude can read the result. + +```bash +sudo /home/gps3/repos/movefaults_clean/scripts/sudo/