fix(monitor): measure descendant CPU; untrack config; retune disk thresholds

CHANGELOG.md

@@ -7,6 +7,42 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Fixed
+
+- **Monitor reported 0.0% CPU for every child process.** psutil records the
+  CPU-times baseline on the `Process` *instance*, and `children(recursive=True)`
+  builds fresh objects on every sample, so each descendant's first (always-zero)
+  reading was the only one ever taken. Process instances are now cached by PID,
+  so baselines persist. This mattered whenever the real work ran in a child
+  rather than the launched process — a Windows venv `python.exe` shim, or
+  EddyPro spawning workers — which produced exactly the all-zero CPU column the
+  process-tree fix was meant to eliminate.
+
+- **`eddypro-batch` with no subcommand printed a config error instead of
+  help.** `main()` validated the config file before dispatching, so on a fresh
+  clone -- where `config/config.yaml` does not exist yet -- the first command
+  anyone runs failed with "Configuration file not found" rather than showing
+  usage. The no-command case now short-circuits to `print_help()`.
+
+- **Pre-v2 metrics files are now reported as `UNKNOWN`** instead of being read
+  as a run with no bottleneck. Those files lack every process-tree column, so
+  the analyser was summarising absent data as `CPU 0.0%` and concluding there
+  was headroom to spare.
+
+### Changed
+
+- **`config/config.yaml` is no longer tracked in git.** It holds
+  machine-specific absolute paths that churned in every diff. The file stays
+  where it is and the default `--config` path is unchanged; only
+  `config/config.yaml.example` is version-controlled now, so a fresh clone must
+  copy it before first use.
+
+- **Default `performance_thresholds` disk limits raised to SATA SSD scale**
+  (`disk_high_mb_per_s` 100 → 450, `disk_moderate_mb_per_s` 50 → 250,
+  `disk_high_iops` 1000 → 20000). The previous values assumed a mechanical disk
+  and flagged ordinary SSD runs as disk-bound. `docs/CONFIG.md` now carries a
+  per-medium table for NVMe, SATA, and mechanical drives.
+
 ### Added
 
 - **`--version` global CLI flag** to print the installed package version and exit.

CLAUDE.md

@@ -51,7 +51,9 @@ pre-commit run --all-files
   `D:/L0_raw/`, or `D:/L1_processed/`. These hold irreplaceable raw and
   processed field data.
 - `config/config.yaml` is the user's live working config with
-  machine-specific paths. Don't commit changes to it.
+  machine-specific paths. It is untracked and gitignored — never re-add it
+  with `git add -f`. Edit `config/config.yaml.example` instead when a change
+  belongs in the repo.
 
 ## Code quality
 

README.md

@@ -62,6 +62,9 @@ cp config/config.yaml.example config/config.yaml
 # Edit config/config.yaml with your paths and settings
 ```
 
+This step is required: `config/config.yaml` is gitignored (it holds
+machine-specific paths), so a fresh clone does not contain one.
+
 **Key configuration items:**
 
 - Set `eddypro_executable` to your EddyPro installation path

config/config.yaml.example

@@ -59,12 +59,19 @@ monitoring_enabled: true
 metrics_interval_seconds: 0.5
 
 # Optional: tune how the bottleneck analyser classifies a run. Defaults assume a
-# mechanical disk; raise the disk limits substantially for NVMe. See
-# docs/CONFIG.md for the full table of keys and defaults.
+# SATA SSD (~550 MB/s). Set these to match the drive the input data lives on,
+# otherwise the disk verdict is meaningless. See docs/CONFIG.md for every key.
 #performance_thresholds:
 #  cpu_high_percent: 90
 #  memory_high_percent: 85
-#  disk_high_mb_per_s: 100
+#  # NVMe SSD:
+#  disk_high_mb_per_s: 3000
+#  disk_moderate_mb_per_s: 1500
+#  disk_high_iops: 200000
+#  # Mechanical disk:
+#  #disk_high_mb_per_s: 150
+#  #disk_moderate_mb_per_s: 80
+#  #disk_high_iops: 150
 
 # Reporting configuration
 reports_dir: null  # Optional: Custom reports directory (default: {output_dir}/reports)

docs/CONFIG.md

@@ -8,6 +8,16 @@ Default location: `config/config.yaml`
 
 Override with: `--config /path/to/config.yaml`
 
+`config/config.yaml` is **not tracked in git** — it holds machine-specific
+absolute paths that differ per install. It is covered by `.gitignore`, so a
+fresh clone has no `config/config.yaml` until you create one:
+
+```powershell
+cp config/config.yaml.example config/config.yaml
+```
+
+Only `config/config.yaml.example` is version-controlled.
+
 ## Configuration Schema
 
 ### Complete Example
@@ -408,8 +418,11 @@ monitoring_enabled: false
 **Default:** see table below
 
 **Description:** Tunes how the bottleneck analyser classifies a run. The
-defaults assume a mechanical disk; on NVMe storage the disk limits should be
-raised substantially or every run will be reported as disk-bound.
+defaults assume a **SATA SSD** (~550 MB/s sequential), which is the common case
+for the bulk storage EddyPro reads from. The disk limits must match the drive
+the input data actually lives on: leave NVMe limits at the SATA default and
+ordinary runs are reported as disk-bound; leave mechanical-disk limits at the
+SATA default and a genuinely saturated disk is never flagged at all.
 
 | Key | Default | Meaning |
 |-----|---------|---------|
@@ -418,17 +431,33 @@ raised substantially or every run will be reported as disk-bound.
 | `cpu_idle_percent` | 40 | Below this, a busy disk is read as the limiting factor |
 | `memory_high_percent` | 85 | System memory use that counts as RED |
 | `memory_moderate_percent` | 70 | System memory use that counts as YELLOW |
-| `disk_high_mb_per_s` | 100 | Combined read+write throughput counting as RED |
-| `disk_moderate_mb_per_s` | 50 | Throughput counting as YELLOW |
-| `disk_high_iops` | 1000 | Combined IOPS above which latency is the suspect |
+| `disk_high_mb_per_s` | 450 | Combined read+write throughput counting as RED |
+| `disk_moderate_mb_per_s` | 250 | Throughput counting as YELLOW |
+| `disk_high_iops` | 20000 | Combined IOPS above which latency is the suspect |
 
 Unknown keys are ignored, so a config written for a newer version still loads.
 
+Suggested disk limits by medium:
+
+| Medium | `disk_high_mb_per_s` | `disk_moderate_mb_per_s` | `disk_high_iops` |
+|--------|---------------------|--------------------------|------------------|
+| NVMe SSD | 3000 | 1500 | 200000 |
+| SATA SSD (default) | 450 | 250 | 20000 |
+| Mechanical / USB HDD | 150 | 80 | 150 |
+
+On Windows, check which you have with:
+
+```powershell
+Get-PhysicalDisk | Select-Object FriendlyName, MediaType, BusType
+Get-Partition | Where-Object DriveLetter | Select-Object DriveLetter, DiskNumber
+```
+
 **Example (NVMe):**
 ```yaml
 performance_thresholds:
-  disk_high_mb_per_s: 2000
-  disk_moderate_mb_per_s: 1000
+  disk_high_mb_per_s: 3000
+  disk_moderate_mb_per_s: 1500
+  disk_high_iops: 200000
 ```
 
 **CLI Override:** none — config only.

examples/multi_year_config.yaml

@@ -44,6 +44,13 @@ log_eddypro_output: true
 # Performance monitoring: sample CPU/memory/disk every second per year-worker.
 # Set monitoring_enabled: false for maximum throughput on large batches.
 monitoring_enabled: true
+
+# Bottleneck thresholds must match the drive the data lives on. These are the
+# defaults, sized for a SATA SSD; see docs/CONFIG.md for NVMe and HDD values.
+performance_thresholds:
+  disk_high_mb_per_s: 450
+  disk_moderate_mb_per_s: 250
+  disk_high_iops: 20000
 metrics_interval_seconds: 1.0
 
 # Reports directory set explicitly: it otherwise defaults to the FIRST

src/eddypro_batch_processor/analysis.py

@@ -46,13 +46,24 @@ class PerformanceThresholds(TypedDict, total=False):
     "cpu_idle_percent": 40.0,
     "memory_high_percent": 85.0,
     "memory_moderate_percent": 70.0,
-    # Roughly a mechanical-disk ceiling; raise substantially for NVMe.
-    "disk_high_mb_per_s": 100.0,
-    "disk_moderate_mb_per_s": 50.0,
-    "disk_high_iops": 1000.0,
+    # Calibrated for a SATA SSD (~550 MB/s sequential), which is the common
+    # case for the bulk storage EddyPro reads from. Adjust for other media:
+    #   NVMe SSD  -> disk_high ~3000, moderate ~1500, iops ~200000
+    #   Mechanical -> disk_high ~150,  moderate ~80,   iops ~150
+    "disk_high_mb_per_s": 450.0,
+    "disk_moderate_mb_per_s": 250.0,
+    "disk_high_iops": 20000.0,
 }
 
 
+# Columns the process-tree monitor must emit. Their absence means the file was
+# written by the pre-v2 monitor, whose figures were all 0.0 anyway (it sampled
+# the shell wrapper rather than EddyPro). Reporting UNKNOWN is honest; reporting
+# "no bottleneck, CPU 0.0%" from those files is not.
+CANONICAL_COLUMNS = ("cpu_percent", "memory_mb", "read_mb", "write_mb")
+LEGACY_COLUMNS = ("process_cpu_percent", "process_memory_rss", "process_io_read_bytes")
+
+
 @dataclass
 class MetricStats:
     """Summary statistics for one metric series."""
@@ -170,6 +181,24 @@ def analyze_rows(
         self, rows: list[dict[str, Any]], scenario_name: str = "baseline"
     ) -> ScenarioAnalysis:
         """Analyse already-parsed metric rows."""
+        if rows and not any(c in rows[0] for c in CANONICAL_COLUMNS):
+            legacy = any(c in rows[0] for c in LEGACY_COLUMNS)
+            detail = (
+                "written by a pre-v2 monitor, whose process metrics were all 0.0"
+                if legacy
+                else "missing every process-tree column"
+            )
+            logger.warning(
+                f"Metrics for {scenario_name} are {detail}; cannot classify."
+            )
+            return ScenarioAnalysis(
+                scenario_name=scenario_name,
+                sample_count=len(rows),
+                explanation=(
+                    f"Unrecognised metrics schema ({detail}). Re-run with the "
+                    f"current version to get a bottleneck verdict."
+                ),
+            )
 
         def series(key: str) -> list[float]:
             return [v for v in (_to_float(r.get(key)) for r in rows) if v is not None]

src/eddypro_batch_processor/cli.py

@@ -1327,25 +1327,29 @@ def main() -> int:
     # Set up logging early
     setup_logging(args.log_level)
 
+    # With no subcommand there is nothing to configure, so help must print
+    # rather than a config error. On a fresh clone config/config.yaml does not
+    # exist yet, and a bare `eddypro-batch` is the first thing anyone runs.
+    if not args.command:
+        parser.print_help()
+        return 1
+
     # Validate config file exists if provided
     if hasattr(args, "config") and args.config:
         config_path = Path(args.config)
         if not config_path.exists():
             logging.error(f"Configuration file not found: {config_path}")
             return 1
 
-    # Route to appropriate command handler
-    if args.command == "run":
-        return cmd_run(args)
-    elif args.command == "scenarios":
-        return cmd_scenarios(args)
-    elif args.command == "validate":
-        return cmd_validate(args)
-    elif args.command == "status":
-        return cmd_status(args)
-    else:
-        parser.print_help()
-        return 1
+    # Route to appropriate command handler. argparse restricts args.command to
+    # these choices and the empty case returned above, so there is no fallback.
+    handlers = {
+        "run": cmd_run,
+        "scenarios": cmd_scenarios,
+        "validate": cmd_validate,
+        "status": cmd_status,
+    }
+    return handlers[args.command](args)
 
 
 if __name__ == "__main__":

src/eddypro_batch_processor/monitor.py

@@ -123,6 +123,12 @@ def __init__(
         self._io_baseline: tuple[float, float, float, float] | None = None
         self._primed_pids: set[int] = set()
 
+        # psutil records the CPU-times baseline on the Process *instance*, so a
+        # freshly constructed object always reports 0.0. children() builds new
+        # objects on every call, so the instances must be cached by PID or every
+        # descendant reads as idle for the whole run.
+        self._proc_cache: dict[int, psutil.Process] = {}
+
         # Previous sample state, for delta-based rate computation
         self._prev_time: float | None = None
         self._prev_io: tuple[float, float, float, float] | None = None
@@ -285,10 +291,26 @@ def _iter_tracked(self) -> list[Any]:
         if not self._process:
             return []
         try:
-            return [self._process, *self._process.children(recursive=True)]
+            children = self._process.children(recursive=True)
         except Exception:
             return []
 
+        tracked: list[Any] = [self._process]
+        live_pids = {self._process.pid}
+        for child in children:
+            # setdefault keeps the first instance seen for this PID so its CPU
+            # baseline survives; the newly built object is discarded.
+            tracked.append(self._proc_cache.setdefault(child.pid, child))
+            live_pids.add(child.pid)
+
+        # Drop instances for processes that have exited. Their cumulative I/O is
+        # already retained separately in _io_by_pid, so nothing is lost.
+        for pid in self._proc_cache.keys() - live_pids:
+            del self._proc_cache[pid]
+            self._primed_pids.discard(pid)
+
+        return tracked
+
     def _collect_sample(self) -> dict[str, Any] | None:
         """
         Collect a single performance sample.