fix: repair performance monitoring, run manifest, and multi-year docs - #15
Merged
Merged
Conversation
Housekeeping found during a structural audit, all left over from the migration away from the monolithic eddypro_batch_processor.py script. - Remove tests/.coverage, a binary coverage artifact that was tracked in git despite .gitignore listing .coverage. Add **/ patterns so nested coverage and mypy caches are ignored too. - Drop src/__init__.py. It made `src` itself an importable package, which is wrong under a src-layout with hatchling, and its docstring still described the package as "refactoring from monolithic script". - Remove the mypy and bandit `src/eddypro_batch_processor\.py` exclusions from pre-commit; that path is the old monolith and no longer exists, so the exclusions were dead. - Stop running both `ruff format --check` and `black --check` in CI. The two formatters can disagree and the build had no tiebreaker; black is what pre-commit enforces, so keep black and use ruff for linting only. - Derive __version__ from importlib.metadata. pyproject declared 0.3.0 while __init__.py hardcoded 0.1.0; deriving it makes the skew impossible to reintroduce. - Rewrite requirements.txt to match pyproject. It pinned requests, urllib3, certifi, idna and charset-normalizer (none of which are imported anywhere) while omitting tqdm and pandas, which are real dependencies. - Drop shebangs from ecmd.py and scenarios.py; these modules are imported, never executed directly. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018hHYmK3AS1NjyeSJiJK4uj
generate_scenario_suffix existed in both ini_tools.py and scenarios.py and
produced *different* output for the same input: ini_tools sorted the keys
alphabetically ("_det0_rot1_spk0_tlag2") while scenarios.py used the
canonical rot/tlag/det/spk/hf order ("_rot1_tlag2_det0_spk0").
Only the scenarios.py version is used in production. The ini_tools copy was
dead outside its own tests, but because both were tested the suite actively
enshrined two incompatible suffix formats, so a future caller picking the
wrong import would have silently produced mismatched output directory names.
Delete the ini_tools copy and its tests; scenarios.generate_scenario_suffix
remains the single implementation.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018hHYmK3AS1NjyeSJiJK4uj
The monitor reported 0.0 for CPU and disk I/O on every run. Verified against a real 2.7-hour GL-ZaF run (18,818 samples): process_cpu_percent min/max/mean all 0.0, process_io_read_bytes and write_bytes all 0.0, and process_memory_rss averaging 4.9 MB. Root cause: the monitor was pointed at a single PID, which (see the following commit for the launch-side fix) was the cmd.exe wrapper rather than EddyPro. 4.9 MB RSS is cmd.exe's footprint exactly. Changes: - Sample a process TREE. Every sample re-walks root.children(recursive=True) and aggregates CPU, RSS and I/O across it, so EddyPro workers spawned mid-run are picked up. - Add attach_process(), so a caller can start the monitor and then point it at a subprocess. This replaces the stop_monitoring()/start_monitoring(pid) dance, which wrote a one-sample CSV and summary JSON before overwriting them at the real stop. - Prime cpu_percent() both system-wide and per process. The first call after construction always returns 0.0 because there is no previous measurement to diff against, so the first sample was guaranteed meaningless. - Derive disk RATES. psutil's counters are monotonic since boot, so the old output reported since-boot totals and then took min/max/mean/p95 over them, which is not a meaningful statistic. Now emits read_mb/write_mb (cumulative since monitoring started) plus read_mb_per_s, write_mb_per_s, read_iops and write_iops derived from deltas over the ACTUAL elapsed time between samples, not the nominal interval, which drifts. - Never disable monitoring on a transient error. A single AccessDenied used to null self._process permanently, silently dropping all process columns for the remainder of the run. Errors are now handled per process. - Emit a fixed canonical column set (METRICS_FIELDNAMES) whose first columns are exactly what report.py reads, so the CSV is never ragged. - Report CPU two ways: cpu_percent normalised by core count (comparable with system_cpu_percent, and the right basis for "is the machine saturated"), and cpu_percent_of_core un-normalised, where 100 means one core fully busy. EddyPro is largely single-threaded, so the latter is what reveals a pegged core on a many-core machine. - Guard _samples with a lock (sampler thread vs CSV writer) and raise the join timeout above one sampling period. - Fix the "Duration: 0.00s" log line, which read summary["duration_seconds"] when the value lives under summary["timing"]. - Hold cumulative totals after the tree exits rather than resetting them to zero, so read_mb/write_mb stay monotonic as consumers chart them. Verified on a real subprocess: 96.3% of one core, real RSS, 716 MB read and ~200 MB/s sustained, where every one of those figures was previously 0.0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018hHYmK3AS1NjyeSJiJK4uj
Adds the analysis layer described in docs/plan/PERFORMANCE_ANALYSIS_DESIGN.md and repairs the manifest defects found alongside it. New src/eddypro_batch_processor/analysis.py: - BottleneckAnalyzer digests a metrics CSV into mean/max/p95 per resource and classifies the run as CPU, MEMORY, DISK_THROUGHPUT, DISK_IOPS or NONE, with a RED/YELLOW/GREEN status per resource and a plain-language explanation. - Sustained load (p95) drives the verdict rather than peaks. A saturated CPU outranks a busy disk; a busy disk paired with an idle CPU means the run was waiting on storage. - Thresholds are tunable via a performance_thresholds config block. Defaults assume a mechanical disk and should be raised for NVMe. - Stdlib only, no pandas, so the reporting path stays import-light. report.py, charts: - load_metrics_from_csv read cpu_percent/memory_mb/read_mb/write_mb while the monitor wrote system_cpu_percent/process_memory_rss in bytes. Every lookup missed and defaulted to 0, so the HTML charts were flat zero lines regardless of what had been sampled. This was known issue #3. Coercion is now per field so one blank cell cannot abandon the rest of the row. - Add a Performance Health Check table (traffic lights + primary bottleneck) and a Per-Year Results table to the HTML report. report.py, manifest: - config_checksum used str(hash(json.dumps(...))). Python's hash() is salted per process by PYTHONHASHSEED, so the SAME config produced a DIFFERENT checksum on every run, making the field useless for comparing runs. Now a real SHA256 over the canonicalised config. A correct sha256 helper already existed in this module but was never called outside tests. - Write the manifest atomically (tmp file + os.replace). It was written straight to the destination, so a crash mid-write left truncated JSON that had already destroyed the previous good manifest. cmd_status already caught JSONDecodeError, i.e. the code anticipated corruption instead of preventing it. - Archive a per-run copy under manifests/run_manifest_{run_id}.json, since the canonical filename is fixed and consecutive runs overwrote each other's provenance. - Add manifest_schema_version, a status field, a years[] array carrying per-year outcome including failures, and an errors[] list. - Add get_provenance()/get_git_provenance() recording git SHA, dirty flag, branch, tool version, argv and the EddyPro executable path plus checksum. README and REPORTING.md have long claimed git SHA and EddyPro version were captured; the provenance parameter existed but no caller ever passed it, so the key never appeared in any manifest. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018hHYmK3AS1NjyeSJiJK4uj
…processing The launch-side half of the monitoring fix, plus the config/CLI surface and the execution features the docs already advertised. Launch path (the root cause of the all-zero metrics): - run_subprocess_with_monitoring now takes an argv list and runs WITHOUT shell=True. Under a shell on Windows, Popen spawns `cmd.exe /c ...`, so Popen.pid is the wrapper's PID and EddyPro is only its child. That PID was handed to the monitor, which therefore sampled an idle shell for the entire run. Everything downstream of this was measuring cmd.exe. - Use monitor.attach_process() instead of stop/restart, so the metrics files are written once at the end rather than truncated at the start. Monitoring toggle (implements docs/plan/MONITORING_TOGGLE_IMPLEMENTATION_PLAN): - New monitoring_enabled config key, default true, threaded through run_eddypro_with_monitoring, run_single_scenario and run_scenario_batch. When false no monitor is created and no metrics files are written. - New --monitor/--no-monitor flags on both run and scenarios. - validation.py type-checks the key and only requires a positive metrics_interval_seconds when monitoring is actually enabled. - Fix an override bug while here: --metrics-interval defaulted to 0.5, which is truthy, so `if args.metrics_interval:` always fired and metrics_interval_seconds from config.yaml was silently overwritten on every run. The default is now None and scenarios honours config too. Multiprocessing (previously validated and advertised but never applied): - Extract the per-year body into module-level process_year(), which takes a plain dict so it is picklable, and dispatch years over a ProcessPoolExecutor when multiprocessing is enabled. Worker count is capped by max_processes, the number of years, and os.cpu_count(). - Results are reordered deterministically; completion order is not stable. - process_year never raises: failures become records, so one bad year cannot abort the run and, crucially, stays visible in the manifest. - Output streaming is suppressed while running in parallel, since interleaved stdout from several EddyPro processes is unreadable. Output still reaches the log file. - Workers re-establish logging, which a fresh process does not inherit. Manifest and reporting: - Report generation no longer sits behind `if years_processed:`. A run in which every year failed is exactly the run whose manifest matters most, and previously produced no manifest at all. - Per-year records, run-level errors, provenance and the bottleneck summary are passed into the manifest; timestamps are UTC with an offset instead of naive local time. - scenarios now generates the per-scenario and aggregate HTML reports that docs/SCENARIOS.md promised, and emits scenario_name (prefixed with the year, so entries stay unambiguous across years) from both the run and scenarios paths. cmd_status read scenario_name while scenarios wrote only scenario_index/scenario_suffix, so every scenario showed as "unknown". - A failing scenario now leaves a manifest behind. The error metadata was built in the exception handler and then discarded, because the only json.dump was on the success path. - cmd_status read manifest["outputs"] when the key is "output_dirs", so the Output Directories section never rendered. It now also prints the per-year table and the bottleneck verdict. - Add a global --version flag, which README already told users to run, and --reports-dir on scenarios, which docs/SCENARIOS.md already documented. tests/test_analysis.py adds an intentionally un-mocked test that runs a real workload under the real monitor. Every existing monitor test mocks psutil wholesale, which is why this class of bug survived in CI; that test asserts CPU and disk figures are non-zero and would fail if the shell=True regression returned. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018hHYmK3AS1NjyeSJiJK4uj
The repo had no CLAUDE.md and no .claude/ directory; the only agent guidance
was .github/copilot-instructions.md, and git operations were unguarded.
- CLAUDE.md: project brief loaded each session. Covers the .venv (a stale
second venv/ also exists, which is a live footgun), the test and lint
commands, the module map, and the domain rules that matter most here:
EddyPro runs take hours so never launch one speculatively, and never write
to data/, logs/, D:/L0_raw/ or D:/L1_processed/, which hold irreplaceable
field data. Defers to copilot-instructions.md for code-quality rules rather
than duplicating them.
- Skills: git-commit (deliberate staging, Conventional Commits matching this
repo's history, never --no-verify, never stage config/config.yaml or data),
git-branch-pr (branch naming, never commit to main, rebase vs merge, gh pr
create with the repo template, CI green before merge, --force-with-lease
only), and release (bump pyproject only, since __version__ is now derived).
- Hooks, wired in .claude/settings.json:
- pretooluse_guard.py blocks force-push, any push to main, commits on main,
--no-verify, reset --hard, clean -fdx, filter-branch, staging protected
paths, and Edit/Write under the protected data directories. Chained
commands are split and checked segment by segment, and matching is on git
subcommand boundaries so `echo "my data is fine"` is not a false positive.
- posttooluse_format.py runs black and ruff --fix on just the edited file
and reports anything left over as advisory feedback.
Verified with 15 payload cases covering each blocked operation, the allowed
counterparts, and the false-positive case.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018hHYmK3AS1NjyeSJiJK4uj
The docs were extensive but had drifted from the code, and the project's actual daily use case -- running several years with one set of settings -- was effectively undocumented and easily confused with scenario runs. New docs/MULTI_YEAR_RUNS.md, the headline addition. Opens by distinguishing the two concepts, because they were conflated throughout: a multi-year run applies the SAME settings across several years, while a scenario run applies several combinations of EddyPro processing parameters to the same years. Then a fully worked GL-Dsk 2020-2025 example given both as a config file (new examples/multi_year_config.yaml) and as pure CLI, plus dry-run guidance, the --no-monitor variant, what lands where, how to read the bottleneck table, and the caveats (years are independent, a failed year is skipped and the run continues, reports_dir defaults to the first year's output directory). Corrections, each verified against the source: - README and REPORTING.md claimed provenance capture of git SHA and EddyPro version. Nothing invoked git and the provenance parameter was never passed. Now accurate for what is actually recorded; no input-checksum claims. - REPORTING.md described config_checksum as "SHA256 of config file content". It was str(hash(...)). Now describes what is really computed. - USAGE.md showed sample output with checkmark and cross glyphs; the code emits ASCII [PASS]/[FAIL]/-. - CONFIG.md said years run in parallel with multiprocessing, then contradicted itself 100 lines later by admitting it was not implemented. Multiprocessing is now real, so the contradiction is removed. - CONFIG.md and ARCHITECTURE.md claimed plotly falls back to SVG with a warning; the code only logs at debug and renders without charts. - README linked docs/plan/IMPROVEMENT_PLAN.md, which lives under plan/implemented/. - KNOWN_ISSUES_AND_TODO.md rewritten: issues 3-6 are fixed by this branch; issues 1, 2 and 7 remain and are stated accurately. De-duplication: the full config example appeared five times with divergent values. config/config.yaml.example is now the single source of truth and the others show a short excerpt and link to it. Scenario CLI examples are owned by docs/SCENARIOS.md. Also documents the new monitoring_enabled and performance_thresholds keys, the --monitor/--no-monitor, --version and scenarios --reports-dir flags, and moves the two now-implemented plans into docs/plan/implemented/. All relative links checked and resolving; all documented CLI invocations verified to parse; all example configs verified to validate. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018hHYmK3AS1NjyeSJiJK4uj
CI installed `ruff>=0.1.0` and `black>=23.0.0`, i.e. whatever was newest at build time, while pre-commit pinned ruff v0.1.8 and black 23.12.1. The two had drifted years apart: CI was resolving ruff 0.16.3 and black 26.5.1, so the hooks passed locally and the same code failed in CI. Two concrete breakages, both of which also affect main: - ruff promoted PLR0917 (too many positional arguments) out of preview. It fires 7 times, on functions that already existed before this branch. Added to the ignore list next to PLR0913, which is ignored for exactly the same reason: these are configuration-heavy functions in scientific code, and they are keyword-called at every call site. - black's 26.x stable style changed how it wraps parenthesized assignments and multiline string arguments. black 23 and black 26 reformat the same code in opposite directions, so with the old pins the two would have fought forever. Fixes: - Bound ruff and black in the dev extra so CI cannot silently pick up a formatter release that disagrees with the hooks. - Bump the pre-commit revs to the same versions CI now installs, so local and CI agree by construction, and switch to the non-deprecated `ruff-check` hook id. - Reformat under black 26.5.1. mypy is deliberately left at 1.8.0 in pre-commit: the CI mypy step is `continue-on-error`, so it cannot break the build, and bumping to 2.x is a larger change than belongs in this branch. Verified by installing ruff 0.16.3 and black 26.5.1 locally and running both checks, rather than by pushing and waiting for CI. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018hHYmK3AS1NjyeSJiJK4uj
The un-mocked monitoring test asserted non-zero disk READS. That holds on Windows but not on Linux, where io_counters().read_bytes counts only bytes actually fetched from the storage layer: the burner reads back a file it just wrote, which is served from the page cache, so zero reads is the correct reading rather than a monitoring failure. Assert on combined read+write instead, for both the cumulative totals and the derived rates. Writes are fsync'd, so the combined figure is non-zero on every platform. The CPU and memory assertions are unchanged and already passed on the Linux runners; those are the ones that actually guard against the monitor being pointed at a shell wrapper instead of the real workload. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018hHYmK3AS1NjyeSJiJK4uj
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Repairs the performance monitor (which reported
0.0for CPU and disk onevery run), adds bottleneck analysis and a monitoring on/off switch, fixes the
run manifest's provenance and durability, wires up the multiprocessing that
was advertised but never applied, adds Claude Code workflow tooling with
safety hooks, and corrects documentation that had drifted from the code.
Problem
Five separate issues, found during a structural audit:
2.7-hour GL-ZaF run (18,818 samples):
process_cpu_percentmin/max/meanall
0.0, disk read and write all0.0, andprocess_memory_rssaveraging 4.9 MB.
CLAUDE.md, no.claude/, nothing preventing a force-push or a write into the raw-datadirectories.
run, failed years disappeared from it, and it was only written on success.
times with divergent values. Multi-year runs — the actual daily use case —
were undocumented and conflated with scenario runs.
multiprocessing/max_processesdid nothing. Validated, surfaced as--mp/--max-proc, documented, and then ignored; runs were alwayssequential.
Approach
The monitoring failure had two independent causes, either of which alone
would have zeroed the output:
a) The wrong process was measured.
core.pylaunched EddyPro withsubprocess.Popen(command, shell=True). On Windows that spawnscmd.exe /c "<exe>" ..., soPopen.pidis the shell wrapper's PID, noteddypro_rp.exe. That PID was handed to the monitor, which faithfully sampledan idle shell for the entire run. The 4.9 MB RSS above is cmd.exe's footprint
exactly. Fixed by passing an argv list (no shell) and sampling the whole
process tree, so EddyPro workers are included too.
b) The report read columns that were never written.
load_metrics_from_csvdid
float(row.get("cpu_percent", 0))while the monitor wrotesystem_cpu_percent/process_memory_rssin bytes. Every lookup missed anddefaulted to
0, so the charts were flat zero lines regardless of what hadbeen sampled. (Known issue #3.)
Also fixed in the monitor: disk I/O was only ever stored as since-boot
cumulative counters, so no MB/s figure existed anywhere and the summary took
the mean of a monotonic counter;
cpu_percent()was never primed, guaranteeinga meaningless first sample; and a single transient
AccessDeniedpermanentlynulled the tracked process, silently dropping all process columns for the rest
of the run.
Changes Made
monitor.pyrewritten: process-tree sampling,attach_process(),primed CPU counters, real MB/s and IOPS derived from deltas over actual
elapsed time, per-process error isolation, a locked sample buffer, and a
fixed canonical column set whose first columns are exactly what
report.pyreads. CPU is reported both core-normalised and un-normalised
(
cpu_percent_of_core, where 100 = one core busy) because EddyPro islargely single-threaded.
analysis.py(new):BottleneckAnalyzerclassifies a run as CPU,MEMORY, DISK_THROUGHPUT, DISK_IOPS or NONE with traffic-light statuses and a
plain-language explanation, driven by sustained load (p95) rather than
peaks. Thresholds are tunable via
performance_thresholds.monitoring_enabledconfig key and--monitor/--no-monitoronrunandscenarios.process_year()and dispatched over aProcessPoolExecutor; streaming issuppressed in parallel mode since interleaved EddyPro output is unreadable.
manifest_schema_version, ayears[]array that keeps failed yearsvisible, UTC timestamps, and a populated
provenanceblock (git SHA/dirty,tool version, EddyPro path + checksum, argv).
docs/SCENARIOS.mdpromised..claude/+CLAUDE.md: project brief, three skills, and hooks thatblock force-push, pushes/commits on
main,--no-verify,reset --hard,and any write under
data/,logs/,D:/L0_raw,D:/L1_processed.tests/.coverage, the vestigialsrc/__init__.py, dead pre-commit exclusions pointing at the deletedmonolith, the duplicated
generate_scenario_suffix(whose two copiesproduced different output), the version skew between
pyprojectand__init__, and CI running bothruff formatandblack.docs/MULTI_YEAR_RUNS.mdwith a worked GL-Dsk 2020-2025example as both config and CLI, opening by disambiguating multi-year runs
from scenario runs.
Tests
196 passing, up from 182; coverage 70.2% to 75.8%.
The key addition is
tests/test_analysis.py, which includes an intentionallyun-mocked test that runs a real workload under the real monitor. Every
pre-existing monitor test mocks
psutilwholesale, which is precisely whythis class of bug survived in CI for so long. It asserts CPU and disk figures
are non-zero and would fail if the
shell=Trueregression returned. It alsocaught a further real defect during development: cumulative counters reset to
zero after the process exited.
Measured on a real subprocess — every figure was previously
0.0:Manual end-to-end verification: config checksum confirmed stable across two
identical runs; a total-failure run now writes a manifest where it previously
wrote none;
statusrenders the per-year table and Output Directories section(the latter never rendered before due to an
outputsvsoutput_dirskeybug); multiprocessing measured at 1.45x on two years (bounded by pool startup —
hour-long real runs will approach Nx); every documented CLI invocation parses;
all example configs validate; 0 broken doc links.
Risks
shell=Truechanges how EddyPro is invoked. This is the corefix, but it means argument quoting is now handled by the OS rather than the
shell. Covered by
tests/test_core.py, which asserts the command is an argvlist and carries the project file.
(
multiprocessing: false), so nothing changes unless explicitly enabled.Years write to separate output directories, so they do not contend.
metrics_*.csvfiles will not loadinto the new charts. Since every value in them was
0.0, nothing of valueis lost.
worth knowing before the first force-push attempt.
ruff/black were newest while pre-commit pinned versions years older, so the
hooks passed locally and the same code failed in CI. This was already true
on
main: ruff 0.16.3 promoted PLR0917 out of preview and it fires 7 timeson pre-existing functions. Bumping these now requires updating
pyproject.tomland.pre-commit-config.yamltogether, which is the point.Rollback Plan
Each commit is independently revertable and scoped to one concern. To disable
just the new monitoring behaviour without reverting code, set
monitoring_enabled: false(or pass--no-monitor); to disable parallelism,set
multiprocessing: false. Revertingfeat(cli,core)alone restores theprevious execution path.
Checklist
Note
config/config.yamlwas deliberately left uncommitted; it holds your livemachine-specific paths. Separately, it is tracked despite
.gitignore'sconfig/*intent, so those paths will keep appearing in diffs —git rm --cached config/config.yamlwould stop that, but it changes yourworking setup so I left the call to you.
🤖 Generated with Claude Code
https://claude.ai/code/session_018hHYmK3AS1NjyeSJiJK4uj