From 597f67774e9cc3ba638fa73b9009c3bb279dfa83 Mon Sep 17 00:00:00 2001 From: Brendan Griffen Date: Mon, 27 Jul 2026 21:43:05 +1000 Subject: [PATCH 01/19] docs: rewrite README and add codebase audit The README described the project as ready to use ("Not a single line of code is required by the user") while omitting that it cannot run: the `modules` I/O package it depends on is absent from the repository, the code is Python 2 only, and it targets the `enthought.*` Traits namespace retired in 2011. Rewrite it to be accurate and useful: - state the archived status and the Python 2 / ETS 3.x constraint up front - document the missing `modules` dependency and what each import provides - list the dependency set reverse-engineered from the imports, since there is no manifest - record the run-directory naming convention, which is encoded implicitly in 18 places in the source and nowhere else - fix the broken screenshots/inspectparams.png link and surface the four screenshots that were present but unreferenced - add MUSIC / Gadget / Rockstar / consistent-trees citations Add AUDIT.md recording the findings behind that rewrite: the blocking issues, 18 correctness defects with line references, security and robustness concerns, the duplication inventory, and a prioritised plan. --- AUDIT.md | 329 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ README.md | 271 ++++++++++++++++++++++++++++++++++++-------- 2 files changed, 554 insertions(+), 46 deletions(-) create mode 100644 AUDIT.md diff --git a/AUDIT.md b/AUDIT.md new file mode 100644 index 0000000..649d3f7 --- /dev/null +++ b/AUDIT.md @@ -0,0 +1,329 @@ +# CME codebase audit + +Assessment of the repository as of 2026-07-27, covering 11 Python modules (~180 KB, +~4,000 lines) plus 13 screenshots. Line references are to the current `master` (`6067f9f`). + +## Verdict + +CME is a well-scoped piece of scientific tooling — the pipeline it automates is genuinely +intricate, and the tabbed workflow maps onto it sensibly. But as checked out it **cannot +run**, and it has not been able to run on a stock environment for roughly a decade. The +important structural problem is not the age of the dependencies; it is that every piece of +science and every file-format writer is welded to a GUI event handler, so nothing can be +tested, scripted, or reused without a display and a cluster. + +The first decision is not technical: **decide whether this repository is an archive or a +project.** The two paths diverge immediately, and most of the work below is only worth doing +on the second one. Sections 1–5 are the findings; section 6 proposes a reorganisation and +section 7 a prioritised plan. + +--- + +## 1. Blocking issues — the code cannot execute + +| # | Issue | Evidence | +| --- | --- | --- | +| 1.1 | **The `modules` package is missing from the repository.** Every module imports it, directly or via `Common.py`'s star import. There is no vendored copy, submodule, or install instruction. | `Common.py:31-36`, `ics.py:5`, `contam.py:2`, `mergertree.py:2`, `gadgetrun.py:2` | +| 1.2 | **Python 2 only.** 4 of 11 modules fail to parse on Python 3 (`print` statements, `xrange`). | `gadgetrun.py:291`, `halos.py:400`, `ics.py:255`, `install.py:50` | +| 1.3 | **Retired Traits namespace.** The `enthought.*` namespace packages were dropped in ETS 4 (2011). No current release provides `enthought.traits.api` or `enthought.traits.ui.wx.editor`. | `Common.py:5-16` | +| 1.4 | **wxPython backend.** `matplotlib.backends.backend_wxagg` and the `NavigationToolbar2Wx` API used here predate matplotlib 2.x; TraitsUI has since moved to Qt as its practical default. | `Common.py:2-3`, `Common.py:63` | +| 1.5 | **The Install tab builds from a `lib/installs/` tree that is not in the repository.** Every install button `chdir`s into a hardcoded relative path (`./lib/installs/fftw-2.1.5/` etc.) that does not exist. | `install.py:49,95,141,157,170,184` | + +Items 1.1 and 1.5 mean that even a perfect Python 2.7 + EPD environment would not get you a +working application. + +--- + +## 2. Correctness defects + +These are independent of the environment problems — they are bugs in the logic as written. +Grouped by confidence. + +### 2.1 Certain (would raise, or silently do nothing) + +| # | Defect | Location | +| --- | --- | --- | +| 2.1.1 | **SLURM submission raises `TypeError`.** `SLURMcores`, `SLURMtime` and `SLURMmemory` are integer traits concatenated directly with strings. The PBS path is the only known-good one. | `gadgetrun.py:320,323,325` | +| 2.1.2 | **The Gadget parameter sweep never executes.** `__init__` assigns `self.nvir = [...]` but the trait is `nrvir`; Traits silently creates a new attribute, leaving `nrvir` an empty list, so every `for nrviri in self.nrvir` loop body is skipped. | `gadgetrun.py:845` vs `:103` | +| 2.1.3 | **Four state updates are no-ops** — `==` written where `=` was meant, so `makeactive` never changes and the affected controls stay disabled. | `contam.py:272,275,281,284` | +| 2.1.4 | **`gethostid()` raises `UnboundLocalError` when no host matches**, but callers test `if idhost:` as though it returns a falsy value. | `halos.py:130-140`, used at `:188,215` | +| 2.1.5 | **`gethalos_xy()` raises `NameError` on the parent full-box path**, because the `'pos' in halo_varx` block reads `idhost`, which only exists in the zoom branch. The same function returns unbound `x, y` when `dataexists` is false. | `halos.py:220-224`, `:231` | +| 2.1.6 | **`_plot_button_fired` does nothing in parent mode.** The entire plotting block is indented inside the `elif self.parentorzoom == 'zoom'` branch, so the parent branch computes `dataexists` and falls through. | `halos.py:234-285` | +| 2.1.7 | **The Odyssey cluster preset never applies.** `self.clustopt = 'odyssey'` — typo for `clusteropt`; Traits accepts it as a new attribute, so the cluster stays at its default. | `header.py:75` | +| 2.1.8 | **`Clear` corrupts the ICs tab.** `self.main.mergertreetab.initstab = []` sets a stray attribute on the wrong object; the intent was `self.main.initstab.haloid = []`. | `candidates.py:148` | +| 2.1.9 | **Three MUSIC Poisson settings are never applied.** `__init__` writes `self.pre_smooth`, `self.post_smooth` and `self.grad_order`, but the traits are `presmooth`, `postsmooth`, `gradorder`. Compounding this, `presmooth`/`postsmooth` are declared as plain class integers rather than traits, yet the view builds `Item`s for them. | `ics.py:698-699,705` vs `:75-76,78` | +| 2.1.10 | **`nhalo` can be unbound**, if the selected halo ID is absent from the candidate file. | `ics.py:533-535` | +| 2.1.11 | **`determineboolstr` returns unbound** for any input that is not exactly `True` or `False` (e.g. a numpy bool). | `ics.py:889-895` | +| 2.1.12 | **A run script is opened before its directory is known to exist** — the `open(filepath + "/runscript")` sits outside the `else` that guards on the path being found, so a missing path raises `IOError` after having already printed "PATH NOT FOUND". | `gadgetrun.py:290-314` | + +### 2.2 Wrong results (runs, but the output is incorrect) + +| # | Defect | Location | +| --- | --- | --- | +| 2.2.1 | **3D halo distribution plots y twice instead of z**: `z = tmphalos['posY']`. Every full-box Mayavi scene is projected into a plane. | `halos.py:369` | +| 2.2.2 | **Reported host x-position is actually its z-position**: `self.hostposx = host['posZ']`. | `halos.py:201` | +| 2.2.3 | **A phantom zero-mass candidate contaminates every selection.** `candidatearr` is initialised to `np.zeros((1,6))` and results are `vstack`ed onto it, so row 0 is always `[0,0,0,0,0,0]` — it is plotted, counted in the array view, and accepted by the "add halo" membership test. | `candidates.py:314,217,124-125` | +| 2.2.4 | **Lagrangian pointer filenames embed a Python list literal.** `'NRVIR' + str(self.nrvir)` on a `List` trait yields `H190897NRVIR['1']` — visible in `screenshots/constructICs.png`. The actual file is written with `int(self.nrvir[0])`, so the displayed and real paths disagree. | `ics.py:340-341,346-347,352-353` vs `:449,665,675` | +| 2.2.5 | **The SLURM execute line is built from the PBS core count**, so a SLURM job launches with `PBSncores` ranks. | `gadgetrun.py:333` | +| 2.2.6 | **`_SLURMcores_changed` has two identical branches**, so enabling SUBFIND does not switch SLURM to the `_sub` binary and `param_sub.txt` the way the PBS path does. | `gadgetrun.py:433-437` | + +### 2.3 Dead or unreachable code + +- `main.py:98-99` — `_analysistab_default` instantiates `Analysis`, whose import is commented out at `main.py:8`. Not currently reachable, but it is a latent `NameError`. +- `halos.py:57` — `halo_type` offers three values, but four branches are written; `'subs of ID'` can never be selected (`halos.py:180,300,364`). +- `halos.py:54-55` — `boxtype` is declared twice; the second declaration silently wins. +- `gadgetrun.py:226` — the view builds `Item('clusteropt')` but the trait declaration is commented out at `gadgetrun.py:5`; the attribute only appears via assignment at `:927`. +- `candidates.py:44` (`upper`), `:17` (`id_select`), `Common.py:33` (`readids`) are never used. + +--- + +## 3. Security and robustness + +| # | Issue | Detail | +| --- | --- | --- | +| 3.1 | **Command injection / breakage via string-concatenated shell commands.** 70 call sites use `os.system` or `subprocess.call(..., shell=True)` with GUI-supplied paths interpolated in. A path containing a space silently splits; one containing `;` or `$(...)` executes. | `install.py` (all buttons), `ics.py:257,325`, `gadgetrun.py:302,306,309,362,368,370,406` | +| 3.2 | **`rm` commands built from unvalidated paths.** `f1.write("rm " + str(filepath) + "/*.e*\n")` and `rm wnoise* temp*` run in a directory derived from GUI state. If a path trait is empty, the glob widens. | `gadgetrun.py:374-375`, `ics.py:322` | +| 3.3 | **A fragile offset into a third-party file.** `tail -n+96 .../Config.sh` assumes the P-Gadget3 config has exactly 95 lines of preamble. Any upstream change silently corrupts the generated config. | `gadgetrun.py:359,365` | +| 3.4 | **Personal identifiers hardcoded in source** — 19 occurrences across usernames (`bgriffen`, `uqbgriff`), hostnames (`csr-dyn-150.mit.edu`, `bigbang.mit.edu`, `rclogin13.rc.fas.harvard.edu`), absolute paths (`/bigbang/data/`, `/n/home01/bgriffen/data/`, `/Users/griffen/Desktop/cme/`) and email addresses. Also `halos.py:131`, an absolute path baked into a method body. | `header.py:55-99`, `gadgetrun.py:9-10,123,257,263`, `halos.py:131` | +| 3.5 | **Every file handle is unprotected by `with`.** ~15 `open()`/`close()` pairs leak on exception, and `_getcandidates_button_fired` can leave `out` open on any error mid-loop. | `ics.py:666,676,743,837`, `gadgetrun.py:253,312,657,664`, `candidates.py:175` | +| 3.6 | **`open()` used as an existence test**, in a `try`/`except IOError`, rather than `os.path.exists`. Race-prone and obscures intent. | `ics.py:374-378,450-451,515-519`, `gadgetrun.py:813-817` | + +--- + +## 4. Structural problems + +### 4.1 The science is trapped inside event handlers + +Every computation lives inside a `_*_button_fired` or `_*_changed` method that also clears +a figure, sets axis labels, and calls `wx.CallAfter`. There is no function you can call to +select candidates, extract a Lagrangian region, or emit a MUSIC config. Consequences: + +- Nothing can be unit tested — you need a display and a live parent simulation to exercise + a mass cut. +- Nothing can be batch-scripted. A 2,000-halo suite is driven by clicking. +- Bugs like 2.1.6 (a mis-indented plotting block) are invisible, because the logic and the + rendering are one blob. + +This is the single highest-value thing to change, and it is the organising idea behind the +layout proposed in section 6. + +### 4.2 Massive duplication + +| Duplicated thing | Occurrences | Where | +| --- | --- | --- | +| The `H..._B..._Z..._P..._LN..._LX..._O..._NV...` folder name | **18** | `contam.py` ×13, `halos.py` ×2, `gadgetrun.py` ×2, `ics.py` ×1 | +| `dircheck = ... + "/outputs/groups_0" + snapnum` followed by an existence test | 9 | `contam.py:209-284,426` | +| The `if self.projopt == 'xy' / 'xz' / 'yz'` axis-selection block | 3 | `ics.py:391-419,462-490,613-641` | +| The 12-line softening cascade | 2 | `gadgetrun.py:522-533,543-554` | +| The `mpirun -np ... ./P-Gadget3 ...` execute-line builder | 12 | `gadgetrun.py:415-453,928-929` | +| The hardcoded Hubble parameter `0.6711` | 6 | `candidates.py:151,305`, `contam.py:93,316,385`, `halos.py:400` | +| Near-identical `_*_changed` handlers that clear the figure | 4 | `mergertree.py:85-123` | + +`contam.py` is the clearest case: 8 of its 12 methods begin with the same 8-line folder-name +construction followed by the same existence check. Extracting one `run_dir_name()` function +and one `outputs_exist()` predicate removes roughly 150 lines and eliminates the class of bug +where one copy is updated and the others are not. + +The duplicated Hubble constant is a scientific correctness risk, not just untidiness: the +value appears both as a trait (`candidates.py:305`) and as five hardcoded literals, one of +which (`halos.py:400`) is inside a debug `print`. + +### 4.3 Star imports and accidental dependencies + +`Common.py` performs six star imports (`enthought.traits.api`, `enthought.enable.api`, +`matplotlib`, `grifflib`, plus module-level `numpy`/`wx`/`os`) and every tab does +`from Common import *`. Nothing declares what it actually uses. + +The concrete hazard: `ics.py` uses `patches.Rectangle` (`:422,495,646`) without importing +`matplotlib.patches`. It resolves only because `from matplotlib import *` at `Common.py:19` +runs after `matplotlib.pyplot` has been imported at `:1`, which populates +`matplotlib.patches` as a side effect. Reorder those two lines and the ICs tab breaks. + +Separately, `ics.py:5` aliases a project module to `re` — shadowing the standard library +`re` for that module. + +### 4.4 Coupling through `self.main` + +Each tab holds a back-reference to `ApplicationMain` and reaches through it to mutate +siblings — `candidates.py:135-139` writes into five other tabs. This makes the initialisation +order load-bearing (`main.py:78-105` must construct `headertab` before anything that reads +`headertab.datamasterpath`, and `candidatestab` before the four tabs that copy its `haloid` +list) and there is no mechanism enforcing it. Traits' own notification system (`@on_trait_change`, +or a shared model object that tabs observe) is the idiomatic fix. + +Note also that the halo sample is *copied* at construction time (`self.haloidlist = +self.main.candidatestab.haloid`), which is why `candidates.py` has to manually push updates +into each tab afterwards. + +### 4.5 Performance + +- **`ics.py:564-601`** re-reads the parent snapshot's entire `POS` and `ID` blocks, plus the + IC file's `POS`/`ID`, once per `nrvir` value. For a 512³ parent that is ~4 GB of I/O per + iteration, when one read outside the loop would serve all of them. +- **`candidates.py:180-209`** rebuilds `np.array(np.float64(MWcand['posX']))` (and five + siblings) inside the per-candidate loop, making the selection O(N²) in the base sample. + Hoisting six lines makes it O(N). +- **`candidates.py:223-237`** re-plots the full candidate set on every loop iteration, because + the plotting block sits inside the `for` loop rather than after it. +- **`candidates.py:217`** grows the result array with `np.vstack` per match — quadratic + copying. Append to a list, `np.array()` once. + +### 4.6 Style inconsistencies + +- `gadgetrun.py` is indented with 2 spaces; every other module uses 4. +- Naming mixes `lowercase` (`haloid`), `camelCase` (`lagrPos`), `PascalCase` (`InitCondFile`, + mirroring Gadget's parameter names — defensible), and `UPPER_CASE` traits (`PMGRID`). +- `if x == True:` throughout, rather than `if x:`. +- Deep view nesting: `halos.py:100-104` reaches eight `HGroup`/`VGroup`/`Group` levels, several + of them redundant single-child wrappers. +- ~200 lines of commented-out code, including whole blocks (`header.py:35-49`, + `gadgetrun.py:491-512`, `ics.py:776-807`, `halos.py:425-431`). Git history serves this + purpose; the comments only obscure the live logic. + +--- + +## 5. Project hygiene + +| Missing | Impact | +| --- | --- | +| `requirements.txt` / `pyproject.toml` | No reproducible environment. The dependency list must be reverse-engineered from imports — which is how the README's list was produced. | +| Any test | No way to verify a refactor. Nothing is currently testable anyway (§4.1). | +| CI configuration | Nothing catches a syntax error on commit. | +| `LICENSE` | All rights reserved by default, so the code is not legally reusable despite the README inviting collaborators. | +| `.gitignore` | `__pycache__`, `*.pyc`, and generated `.conf`/`param.txt`/`rungadget.sh` artifacts are all untracked-but-unignored. `gadgetrun.py:253` writes `rungadget.sh` into the repository root. | +| Docstrings | One in the entire codebase (`main.py:64`). | +| `CHANGELOG` | 30+ commits are "Update README.md"; there is no record of behavioural change. | + +--- + +## 6. Proposed reorganisation + +The layout follows from §4.1: pure computation and pure text generation move out of the GUI, +into modules that can be imported, tested and scripted without a display. + +``` +cme/ +├── pyproject.toml # deps + packaging (replaces "install EPD") +├── README.md +├── AUDIT.md +├── LICENSE +├── .gitignore +├── clusters.toml # cluster profiles — replaces hostname if-chains +├── src/cme/ +│ ├── app.py # was main.py — window assembly only +│ ├── naming.py # run_dir_name(), lagr_paths(), outputs_exist() +│ │ # ← the 18 duplicated folder-name builders +│ ├── cosmology.py # named cosmologies + ONE Hubble constant +│ ├── config.py # loads clusters.toml; no hostnames in code +│ ├── io/ # the former `modules` package, vendored & pinned +│ │ ├── snapshots.py # readsnap, readsnapHDF5 +│ │ ├── halos.py # RSDataReader, readsubf +│ │ └── trees.py # MTCatalogue +│ ├── science/ # NO plotting, NO traits — plain functions on arrays +│ │ ├── candidates.py # select(catalogue, mass_range, zones) -> DataFrame +│ │ ├── lagrangian.py # region(parent, halo, nrvir) -> Region +│ │ ├── contamination.py # profile(catalogue, centre) -> (r, n, m) +│ │ └── softening.py # from_levelmax(boxsize, lmax) -> Softenings +│ ├── writers/ # pure string/file emitters — templates, not f.write chains +│ │ ├── music_conf.py # was ics.py:738-887 +│ │ ├── gadget_param.py # was gadgetrun.py:653-769 +│ │ ├── gadget_config.py # was gadgetrun.py:556-651 +│ │ └── submit.py # PBS + SLURM from one template each +│ ├── shell.py # subprocess wrapper: argument lists, no shell=True +│ └── ui/ # thin HasTraits tabs — bind widgets, call the above +│ ├── common.py # explicit imports; no `import *` +│ ├── home.py install.py candidates.py ics.py +│ └── gadget.py halos.py mergertree.py contamination.py +├── tests/ +│ ├── test_naming.py # 18 duplications collapse to one tested function +│ ├── test_writers.py # golden-file comparison against known-good configs +│ ├── test_candidates.py # synthetic catalogue, known answer +│ └── test_cosmology.py +├── docs/ +└── screenshots/ +``` + +Four changes carry most of the benefit: + +1. **`naming.py`** — one function for the run-directory convention. Removes ~150 lines and + the whole class of drift bugs described in §4.2. +2. **`writers/`** — the config emitters are already almost pure functions; they take scalars + and produce text. Moved out and driven by templates, they become golden-file testable, + which is the only practical defence against a silently malformed `param.txt`. +3. **`science/`** — functions over arrays, with the figure work left in `ui/`. This is what + makes the pipeline scriptable for a large suite, and it is where bugs 2.2.1–2.2.3 would + have been caught. +4. **`clusters.toml`** — cluster profiles as data. Deletes the `platform.node()` if-chains + (`header.py:55-99`) and the hardcoded paths in `gadgetrun.py:257,263`, and makes the tool + usable by someone who is not the author. + +--- + +## 7. Prioritised plan + +**P0 — Decide the repository's status.** Everything else depends on this. + +*If archiving* (a reasonable choice for a 2014 research tool): +1. The README now carries a status banner, documents the missing `modules` dependency, and + records the directory-naming convention. ✔ +2. Add a `LICENSE` so the code is legally reusable. +3. Tag a final release and mark the repository archived on GitHub. +4. Optionally record the science in a short `docs/method.md` — the selection criteria and + contamination metric are the durable contribution, and they currently exist only as + GUI callbacks. + +Stop there. P1 onward is only worth it if the tool is to be used again. + +*If reviving:* + +**P1 — Make it run** (largest effort; nothing else can be verified until this is done) +1. Vendor or pin the `modules` package (§1.1). Without it there is no way to test anything. +2. Python 2 → 3: `pyupgrade`/`2to3` handles `print` and `xrange`; integer division and + `dict.keys()` need review by hand. +3. `enthought.traits.*` → `traits` / `traitsui`; wx → Qt (PySide6) for both TraitsUI and the + matplotlib canvas in `Common.py:41-71`. +4. Add `pyproject.toml` with pinned versions. Verify `pandas`' `.ix` accessor usage + (`halos.py:192,313`, `ics.py:537-541`) — removed in pandas 1.0, replace with `.loc`. +5. Remove the `install.py` tab, or point it at the system package manager. Hand-building + FFTW2 in 2026 is not the right answer, and the source tree it expects is absent anyway. + +**P2 — Extract and test** (do this *before* fixing bugs, so fixes are verifiable) +1. `naming.py` + `test_naming.py`; replace all 18 call sites. +2. `cosmology.py` with a single Hubble constant; replace all 6 literals. +3. `writers/` + golden-file tests, using a known-good `param.txt` and `.conf` from an existing + run directory as the reference. +4. `science/candidates.py` + a synthetic-catalogue test. +5. GitHub Actions running the suite. + +**P3 — Fix the defects in §2**, each with a regression test. Order: 2.1.1–2.1.3 and 2.1.6 +(features that are simply inert), then 2.2.1–2.2.5 (wrong output), then the rest. + +**P4 — Harden** +1. `shell.py`: argument lists, no `shell=True`, no interpolated paths (§3.1–3.3). +2. `clusters.toml`; strip the hardcoded identifiers in §3.4. +3. `with` for every file handle; `os.path.exists` for existence tests. +4. Hoist the loop-invariant reads in §4.5 — the `ics.py` one is a several-GB-per-iteration win. + +**P5 — Polish** +1. `.gitignore`; stop writing `rungadget.sh` into the repository root. +2. Delete the ~200 lines of commented-out code and the dead branches in §2.3. +3. Normalise `gadgetrun.py` to 4-space indentation; run `black` and `ruff` across the tree. +4. Docstrings on the `science/` and `writers/` functions — those are the reusable API. +5. Flatten the redundant single-child view groups. + +--- + +## Appendix: metrics + +| Metric | Value | +| --- | --- | +| Python modules | 11 | +| Lines of Python | ~4,000 | +| Modules that fail to parse on Python 3 | 4 / 11 | +| Shell invocations (`os.system` / `shell=True`) | 70 | +| Duplicated folder-name constructions | 18 | +| Hardcoded personal paths / usernames / emails | 19 | +| Hardcoded copies of the Hubble parameter | 6 | +| Tests | 0 | +| Docstrings | 1 | +| Largest modules | `gadgetrun.py` (49 KB), `ics.py` (47 KB) | diff --git a/README.md b/README.md index bd0e0f4..47c7080 100644 --- a/README.md +++ b/README.md @@ -1,55 +1,234 @@ -CME: Caterpillar Made Easy -=== +
+ +# CME — Caterpillar Made Easy + +An interactive desktop GUI for setting up, running and inspecting cosmological zoom-in simulations. + +![Python](https://img.shields.io/badge/python-2.7-blue) +![Status](https://img.shields.io/badge/status-archived-lightgrey) +![GUI](https://img.shields.io/badge/GUI-Traits%20%2B%20wxPython-orange) +![Viz](https://img.shields.io/badge/3D-Mayavi-8a2be2) +![License](https://img.shields.io/badge/license-none%20specified-red) + +
+ +CME wraps the entire zoom-in resimulation pipeline — candidate halo selection, Lagrangian +region extraction, MUSIC initial conditions, P-Gadget3 parameter files, cluster job +submission, and contamination analysis — behind a single Traits/TraitsUI window, so a whole +suite of resimulations can be configured and launched without writing code. It was built for +the [Caterpillar Project](https://www.caterpillarproject.org) at MIT/Harvard, but the +workflow applies to any suite built on MUSIC + Gadget + Rockstar. + +> [!WARNING] +> **Status: archived / historical.** Last substantive development was 2014. This is +> **Python 2 only** and targets the retired pre-2011 `enthought.*` Traits namespace with a +> wxPython backend — it will not import on Python 3 or on any current ETS release. It also +> depends on an I/O package that is **not included in this repository** +> (see [Prerequisites](#prerequisites)). Treat it as a reference implementation and a record +> of the Caterpillar setup workflow. See [`AUDIT.md`](AUDIT.md) for a full assessment and a +> modernisation plan. + +--- + +## Features + +| Stage | Tab | What it does | +| --- | --- | --- | +| Dependencies | **Install** | Builds FFTW2/3, HDF5, GSL, zlib and szip into a local `lib/` prefix. | +| Candidate selection | **Candidates** | Scans a parent Rockstar catalogue for Milky-Way analogues using a mass range plus three tunable isolation ("exclusion zone") criteria; tabulates and exports the surviving sample. | +| Lagrangian regions | **Initial Conditions** | Traces a candidate's z=0 particles back to the parent ICs, measures the Lagrangian centre/extent/volume, visualises the region in any projection, and writes the MUSIC region point file. | +| Initial conditions | **Initial Conditions** | Generates MUSIC `.conf` files across a grid of resolutions, paddings, overlaps, region shapes and cosmologies, then invokes MUSIC for each combination. | +| Simulation setup | **Gadget** | Exposes every P-Gadget3 `param.txt` and `Config.sh` option, auto-scales gravitational softenings with `levelmax`, and designs the snapshot output list in expansion factor. | +| Progress tracking | **Gadget → Existence** | Tabulates which ICs, snapshots and halo catalogues already exist across the whole parameter sweep. | +| Job submission | **Gadget → Write & Submit** | Writes and submits PBS/Torque or SLURM run scripts, optionally recompiling Gadget first. | +| Halo analysis | **Halo Finder** | Plots any pair of Rockstar quantities for a parent box or a zoom, restricted to hosts, subhaloes or all; renders 3D halo distributions and velocity fields in Mayavi. | +| Merger histories | **Merger Tree** | Plots main-branch evolution of any tracked quantity from `consistent-trees` output. | +| Resolution quality | **Contam.** | Measures low-resolution particle contamination versus radius from the target FOF group, and maps it as a 2D heatmap in any projection or a 3D Mayavi scene. | + +Cosmologies available throughout: WMAP1, WMAP3, WMAP5, WMAP7, WMAP9, PLANCK. + +--- + +## Screenshots + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Candidate selection
Candidate selection		Lagrangian region inspection
Lagrangian region inspection
Constructing ICs across a sweep
Constructing initial conditions		Merger tree inspection
Merger tree inspection
Halo distributions in 3D (Mayavi)
FOF visualisation		Halo velocity field in 3D (Mayavi)
Halo velocity field
Inter-halo relations from Rockstar
Parent catalogue inspection		Contamination versus radius
Radial contamination
Contamination heatmap
Contamination heatmap		P-Gadget3 parameters
Gadget setup
Snapshot output list
Snapshot output list		Existence check across the sweep
Existence check
Submit to PBS or SLURM
Submit jobs
+ +--- + +## Prerequisites + +### Python environment + +Python 2.7 with the Enthought Python Distribution (EPD) or a comparable ETS 3.x stack: + +- `enthought.traits` / `enthought.traits.ui` — the ETS 3.x namespace, **not** the modern `traits` / `traitsui` packages +- `traitsui` — for `ArrayViewEditor` and `TabularEditor` +- `wxPython` — the TraitsUI and matplotlib backend used here +- `mayavi` + `tvtk` — 3D scenes +- `matplotlib` (with the `WXAgg` backend), `numpy`, `pandas` + +There is no dependency manifest in the repository; the list above is derived from +`Common.py` and the per-tab imports. + +### External I/O package (not included) + +Every module imports a `modules` package that must be on `PYTHONPATH`. Without it, nothing +runs. It provides: + +| Import | Provides | +| --- | --- | +| `modules.readsnapshots.readsnap` | Gadget binary snapshot reader (`read_block`) | +| `modules.readsnapshots.readsnapHDF5` | Gadget HDF5 reader (`read_block`, `snapshot_header`) | +| `modules.readsnapshots.readids` | Particle ID reader | +| `modules.readhalos.RSDataReaderv2` | Rockstar catalogue reader (`RSDataReader`) | +| `modules.readhalos.readsubf` | SUBFIND group catalogue reader (`subfind_catalog`) | +| `modules.mergertrees.MTCatalogue` | `consistent-trees` merger tree reader | +| `modules.convertfiles.reWriteIC` | `getBlocks()` — post-processes MUSIC output for Gadget | +| `modules.brendanlib.grifflib` | Helpers: `cosmoconstant`, `getcandidatelist`, `getcentext`, `getlagrxyz`, `drawcircle`, `addsubtitle`, `placenormtext`, `CorrectPos`, `COM` | + +### External codes and data + +- **MUSIC** (Hahn & Abel 2011) — compiled binary, path set on the Home tab +- **P-Gadget3** — source tree at `/lib/P-Gadget3`, compiled from the Gadget tab +- **Rockstar** halo catalogues and **consistent-trees** merger trees for the parent box +- A parent simulation (the Caterpillar 512³, 100 Mpc/h box) with both `outputs/` snapshots and `ics/` +- A PBS/Torque or SLURM cluster, for job submission + +--- + +## Installation + +```bash +git clone https://github.com/bgriffen/cme.git +cd cme + +# The I/O package is a separate dependency — put its parent directory on PYTHONPATH +export PYTHONPATH=/path/to/modules-parent:$PYTHONPATH +``` + +No `pip install` path exists: there is no `setup.py` or `pyproject.toml`, and the package +is run in place from the repository root. + +## Usage + +```bash +python main.py +``` + +The window splits into a plot panel (matplotlib, plus a Mayavi scene on its own tab) on the +left and the workflow tabs on the right. Marker colour, style and size at the bottom left +apply to every 2D plot. Work left to right through the tabs: + +1. **Home** — set your username and the paths to your home directory, MUSIC, P-Gadget3, the + parent simulation and the project data directory. Everything downstream derives from + these. Cluster presets are keyed off `platform.node()`, so an unrecognised machine needs + its paths entered by hand. +2. **Candidates** — either load an existing `candidates.dat` or run the selection, then add + the IDs you want into the working halo sample. That sample propagates to every other tab. +3. **Initial Conditions** — generate and inspect the Lagrangian region per halo and `nrvir` + value, then produce the MUSIC configs and run them. +4. **Gadget** — set parameters, build the output list, then write and submit the run scripts. +5. **Halo Finder / Merger Tree / Contam.** — analyse the results. + +### Run directory naming + +Every resimulation lives at `/halos/H//`, where the folder +name encodes the full parameter set: + +``` +H_B_Z_P_LN_LX_O_NV +``` + +For example, `H190897_BE_Z127_P7_LN7_LX11_O4_NV4` is halo 190897, ellipsoidal region, +starting at z=127, padding 7, levels 7→11, overlap 4, region enclosing 4·R_vir(z=0). + +Lagrangian region files are written to `/ics/lagr/HNRVIR`, each +alongside a `.head` file holding the region centre and extent. -Interactive suite to do the following tasks for the Caterpillar Project being carried out at MIT/Harvard: +## Project structure -* generate list of candidates based on tunable isolation and merger history criteria -* inspect lagrangian region properties (e.g. size, volume, morphology) -* contruct initial conditions for cosmological simulations using MUSIC (Hahn & Abel 2010). -* run cosmological simulations using P-Gadget3. -* examine contamination of low-resolution particles in resimulation across suite -* create heatmaps of contamination in every projection -* do 3D visualisations of halo and particle velocity field and positions -* submit job via both PBS and SLURM submission systems. +``` +cme/ +├── main.py # Application entry point; assembles the tabbed window +├── Common.py # Shared imports and the matplotlib-in-Traits figure editor +├── header.py # Home tab — cluster detection and base paths +├── install.py # Install tab — builds FFTW/HDF5/GSL/zlib/szip +├── candidates.py # Candidates tab — isolation-criteria halo selection +├── ics.py # Initial Conditions tab — Lagrangian regions and MUSIC configs +├── gadgetrun.py # Gadget tab — param.txt, Config.sh, PBS/SLURM submission +├── halos.py # Halo Finder tab — Rockstar catalogue plots and 3D scenes +├── mergertree.py # Merger Tree tab — main-branch evolution plots +├── contam.py # Contam. tab — low-resolution particle contamination +└── screenshots/ # Interface screenshots used in this README +``` + +Each tab is a `HasTraits` subclass instantiated by `ApplicationMain` and handed a reference +to it, so tabs share state (the halo sample, the base paths, the active figure) through the +`self.main` back-reference. -Not a single line of code is required by the user. +## Testing + +There is no test suite, and no CI configuration. Adding one is the first recommendation in +[`AUDIT.md`](AUDIT.md); the pure-computation parts (candidate selection, softening scaling, +config-file writers, folder-name construction) are the natural place to start, since they +can be tested without a display or a cluster. + +## Known limitations + +- Python 2 syntax and the retired `enthought.*` namespace: the code will not import on + Python 3 or on any current ETS release. +- Cluster and user paths are hardcoded in several places (`header.py`, `gadgetrun.py`), + including specific hostnames, usernames and email addresses. Anyone else must edit source. +- Shell commands are assembled by string concatenation and run with `shell=True`, so paths + containing spaces or shell metacharacters will break or misbehave. +- The SLURM script writer concatenates integer traits with strings and raises `TypeError`; + only the PBS path is known-good. +- No dependency manifest, no packaging, no tests. -The packaged utilizes a number of suites which come with EPD distribution (e.g. Traits, Pandas) which is required for the package to work. A number of read modules are not included for Gadget files and the Rockstar halo catalogues. Although it is for the *Caterpillar Project*, it can in principle work on any cosmological simulation suite. +## Acknowledgments -Please contact [brendan.f.griffen@gmail.com](mailto:brendan.f.griffen) if you would like to use this software for your simulation suites. +Initial conditions are generated with **MUSIC** — Hahn & Abel (2011), *Multi-scale initial +conditions for cosmological simulations*, MNRAS 415, 2101. Simulations are run with +**P-Gadget3**, descended from GADGET-2 — Springel (2005), MNRAS 364, 1105. Halo finding uses +**Rockstar** — Behroozi, Wechsler & Wu (2013), ApJ 762, 109 — and merger trees use +**consistent-trees** — Behroozi et al. (2013), ApJ 763, 18. -## Examples +Developed as part of the Caterpillar Project at MIT/Harvard. -#### Candidate Selection -![candidate selection](screenshots/FOFviz.png) +## Contact -#### Inspect & Generate Lagrangian Regions -![inspect lagrangian region](screenshots/lagrangianinspection.png) +Brendan Griffen — [brendan.f.griffen@gmail.com](mailto:brendan.f.griffen@gmail.com) + +## License -#### Inspect Merger Trees -![inspect lagrangian region](screenshots/mergertreeinspection.png) - -#### Examing Halo Distributions Interactively (via Mayavi) -![Examing Halo Distributions Interactively](screenshots/FOFviz.png) - -#### Examing Halo Velocity Field Interactively (via Mayavi) -![Examing Halo Velocity Field Interactively](screenshots/velocityhaloinspection.png) - -#### Examine Contamination Quality -![Examine Contatmination Quality](screenshots/contaminationradial.png) - -#### Examine Contamination Quality (via heatmap) -![Examine Contatmination Heatmaps](screenshots/contaminationheatmap.png) - -#### Specify All Gadget Parameters (temporal resolution etc.) -![Specify All Gadget Parameters](screenshots/specifysnapshotoutput.png) - -#### Investigate inter-halo relations from Rockstar halo catalogue -![Investigate inter-halo relations](screenshots/inspectparams.png) - -#### Run full suites with varied initial conditions -##### (e.g. to determine how resolution effects contamination rate) -![Run full suites with varying initial conditions](screenshots/existencecheck.png) - -#### Submit jobs to cluster via PBS or SLURM -![Submit jobs](screenshots/submitjob.png) +No license file is present, so all rights are reserved by default. If this is meant to be +reusable, add a `LICENSE` — MIT or BSD-3-Clause is conventional for astronomy tooling. From 4863680e5927656e99d5d44fd79dfe1f6536cb1a Mon Sep 17 00:00:00 2001 From: Brendan Griffen Date: Mon, 27 Jul 2026 21:44:06 +1000 Subject: [PATCH 02/19] chore: add .gitignore Nothing was ignored previously, so Python caches sat alongside generated simulation artifacts as untracked noise. The Gadget tab writes its driver script (rungadget.sh) into the working directory and MUSIC leaves noise and scratch files behind, all of which are reproducible from GUI state and belong with the run directory rather than in version control. --- .gitignore | 56 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 56 insertions(+) create mode 100644 .gitignore diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..9f817f6 --- /dev/null +++ b/.gitignore @@ -0,0 +1,56 @@ +# Byte-compiled / cache +__pycache__/ +*.py[cod] +*$py.class + +# Packaging / distribution +build/ +dist/ +*.egg-info/ +.eggs/ + +# Virtual environments +.venv/ +venv/ +env/ + +# Test and lint caches +.pytest_cache/ +.ruff_cache/ +.coverage +htmlcov/ + +# Editor / OS cruft +.DS_Store +*.swp +.idea/ +.vscode/ + +# Generated simulation artifacts. +# The Gadget tab writes its driver script into the working directory +# (see cme/ui/gadget.py), and MUSIC leaves noise and scratch files behind. +rungadget.sh +wnoise_* +temp_* +ics.[0-9]* +ExpansionList* + +# Generated configuration. These are outputs of the Initial Conditions and +# Gadget tabs, reproducible from the GUI state, and belong with the run +# directory rather than in version control. +*.conf +param.txt +param_sub.txt +Config.sh +Config_sub.sh +runscript +runscript_sub + +# Run outputs +outputs/ +OUTPUT +ERROR +OUTPUTsub +ERRORsub +*.o[0-9]* +*.e[0-9]* From de8d5992b592de930597ac459c4521e66e71de09 Mon Sep 17 00:00:00 2001 From: Brendan Griffen Date: Mon, 27 Jul 2026 21:44:07 +1000 Subject: [PATCH 03/19] docs: add MIT license The repository had no license file, leaving it all-rights-reserved by default while the README invited others to use it for their own simulation suites. MIT is the conventional choice for tooling of this kind. --- LICENSE | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 LICENSE diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..dafbf18 --- /dev/null +++ b/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2014-2026 Brendan Griffen + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. From 62810ac37b1f56448087a649069dbc668f3564f1 Mon Sep 17 00:00:00 2001 From: Brendan Griffen Date: Mon, 27 Jul 2026 21:45:39 +1000 Subject: [PATCH 04/19] refactor: port to Python 3 Four of the eleven modules did not parse under Python 3 at all, so the package could not be imported on any currently supported interpreter. - convert 35 `print` statements to function calls, preserving the space-separated multi-argument output - replace `xrange` with `range` (6 sites) - replace the pandas `.ix` accessor with `.loc` (21 sites). `.ix` was removed in pandas 1.0; these indexers are all label-based lookups of Rockstar halo IDs against the catalogue index, so `.loc` is the correct replacement rather than `.iloc` - escape five stray backslashes in ics.py print literals that Python 3.12 reports as invalid escape sequences, keeping the rendered output identical Verified: every module now compiles cleanly with no SyntaxWarnings. Behaviour is otherwise unchanged; this commit is mechanical. --- candidates.py | 2 +- gadgetrun.py | 12 ++++----- halos.py | 10 ++++---- ics.py | 68 +++++++++++++++++++++++++-------------------------- install.py | 26 ++++++++++---------- 5 files changed, 59 insertions(+), 59 deletions(-) diff --git a/candidates.py b/candidates.py index 7e53ec1..7b1771c 100644 --- a/candidates.py +++ b/candidates.py @@ -177,7 +177,7 @@ def _getcandidates_button_fired(self): #xcand = [] #ycand = [] - for i in xrange(0,len(MWcand)): + for i in range(0,len(MWcand)): # CYCLE CANDIDATE i xposi = np.array(np.float64(MWcand['posX']))[i] yposi = np.array(np.float64(MWcand['posY']))[i] diff --git a/gadgetrun.py b/gadgetrun.py index 78a7621..a88bbe6 100644 --- a/gadgetrun.py +++ b/gadgetrun.py @@ -288,9 +288,9 @@ def _subscript_button_fired(self): filepath = self.gadpath + 'halos/H' + str(self.haloid) + '/' + foldername if not os.path.exists(filepath): - print "PATH NOT FOUND:",filepath + print("PATH NOT FOUND:",filepath) else: - print "PATH FOUND:",filepath + print("PATH FOUND:",filepath) self.levelmaxuse = int(self.lmax) self.Omega0,self.OmegaLambda,self.OmegaBaryon,self.HubbleParam,sigma8,nspec = cosmoconstant(self.cosmologylist[0]) ext = self.gadpath + 'halos/H' + str(self.haloid) + '/' + foldername @@ -408,7 +408,7 @@ def _subscript_button_fired(self): #command = "rm " + str(filepath) + "/rungadget.sh" #subprocess.call(';'.join([command]), shell=True) - print "SUBMITTED JOBS!" + print("SUBMITTED JOBS!") def _ENABLE_SUBFIND_changed(self): if self.ENABLE_SUBFIND == True: @@ -480,8 +480,8 @@ def _vizexpz_button_fired(self): snapshotin = np.array(listin.split(',')) listcomp = np.array(self.ExpansionListArr[:,0]) self.ExpansionListArr[:,3] = np.array(len(expfact)*[0]) - for i in xrange(0,len(listcomp)): - for j in xrange(0,len(snapshotin)): + for i in range(0,len(listcomp)): + for j in range(0,len(snapshotin)): if int(listcomp[i]) == int(snapshotin[j]): self.ExpansionListArr[i,3] = 1 @@ -656,7 +656,7 @@ def makeparam(self,filename,ext,includesub): expfilename = ext + '/' + self.OutputListFilename f = open(expfilename,'w') - for i in xrange(0,len(self.ExpansionListArr[:,0])): + for i in range(0,len(self.ExpansionListArr[:,0])): f.write(str(np.float64(self.ExpansionListArr[i,2])) + ' ' + str(int(self.ExpansionListArr[i,3])) + '\n') f.close() diff --git a/halos.py b/halos.py index c38aa12..a9a5164 100644 --- a/halos.py +++ b/halos.py @@ -189,7 +189,7 @@ def gethalos_xy(self): if idhost: hosts = halodata.get_hosts() subs = halodata.get_subs() - host = hosts.ix[int(idhost)] + host = hosts.loc[int(idhost)] subsinhost = halodata.get_subhalos_from_halo(self.haloid) xposhost = float(host['posX']) yposhost = float(host['posY']) @@ -310,7 +310,7 @@ def _quiver_button_fired(self): else: allhalos = halodata.get_hosts() - halotmp = allhalos.ix[int(self.haloid)] + halotmp = allhalos.loc[int(self.haloid)] xposhost = float(halotmp['posX']) yposhost = float(halotmp['posY']) @@ -372,7 +372,7 @@ def _plotxyzrvir_button_changed(self): else: allhalos = halodata.get_hosts() - halotmp = allhalos.ix[int(self.haloid)] + halotmp = allhalos.loc[int(self.haloid)] xposhost = float(halotmp['posX']) yposhost = float(halotmp['posY']) @@ -397,7 +397,7 @@ def _plotxyzrvir_button_changed(self): rvmax = np.array(halodata.data['rvmax'][condnew]) vmax = np.array(halodata.data['vmax'][condnew]) - print mvir,mvir/0.6711,rvirn,rvmax,vmax,xposhost,yposhost,zposhost + print(mvir,mvir/0.6711,rvirn,rvmax,vmax,xposhost,yposhost,zposhost) extent = [xposhost-boxwidth,xposhost+boxwidth,yposhost-boxwidth,yposhost+boxwidth,zposhost-boxwidth,zposhost+boxwidth] self.main.scene.mlab.points3d(x, y, z, rvir/1000,colormap="copper") @@ -438,4 +438,4 @@ def __init__(self, main, **kwargs): self.halo_varx = 'posX' self.halo_vary = 'posY' self.gadpath = self.main.headertab.datamasterpath - self.halopath = self.main.headertab.parentsimpath + 'RockstarData' \ No newline at end of file + self.halopath = self.main.headertab.parentsimpath + 'RockstarData' diff --git a/ics.py b/ics.py index 64cbeca..be09510 100644 --- a/ics.py +++ b/ics.py @@ -252,8 +252,8 @@ def _generate_button_fired(self): self.confstatus = 'Generated halo configuration files.' runmusic = self.musicpath + '/MUSIC ' + confname cding = "cd " + writepath - print "EXECUTING..." - print runmusic + print("EXECUTING...") + print(runmusic) subprocess.call(';'.join([cding, runmusic]), shell=True) re.getBlocks(writepath) @@ -320,8 +320,8 @@ def _generate_button_fired(self): runmusic = self.musicpath + '/MUSIC ' + confname cding = "cd " + writepath rmfiles = "rm wnoise* temp*" - print "EXECUTING..." - print runmusic + print("EXECUTING...") + print(runmusic) subprocess.call(';'.join([cding,runmusic,rmfiles]), shell=True) #cpconvert = "cp ./lib/reWriteIC.py ./lib/convertics.py " + writepath #runconvert = "python convertics.py" @@ -450,7 +450,7 @@ def _makeic_button_fired(self): try: with open(filename): if len(self.nrvir) == 1: - print "Visualising:", 'HALO' + str(self.haloidselect) + 'NRVIR' + str(int(self.nrvir[0])) + print("Visualising:", 'HALO' + str(self.haloidselect) + 'NRVIR' + str(int(self.nrvir[0]))) figure = self.main.display figure.clear() ax = figure.add_subplot(111) @@ -516,7 +516,7 @@ def _makeic_button_fired(self): try: with open(self.candidatefiledir + self.candidatefilename): pass except IOError: - print 'CANT FIND CANDIDATE LIST - CHECK DIRECTORIES!' + print('CANT FIND CANDIDATE LIST - CHECK DIRECTORIES!') return if self.writelagrfile == False and len(self.nrvir) != 1: @@ -530,45 +530,45 @@ def _makeic_button_fired(self): idcand = getcandidatelist(self.candidatefiledir + self.candidatefilename) idcand = idcand[:,0] - for index in xrange(0,len(idcand)): + for index in range(0,len(idcand)): if idcand[index] == idhalo: nhalo = index - rvircand = allhalos.ix[idhalo]['rvir'] - mvircand = allhalos.ix[idhalo]['mvir'] - posXcand = allhalos.ix[idhalo]['posX'] - posYcand = allhalos.ix[idhalo]['posY'] - posZcand = allhalos.ix[idhalo]['posZ'] + rvircand = allhalos.loc[idhalo]['rvir'] + mvircand = allhalos.loc[idhalo]['mvir'] + posXcand = allhalos.loc[idhalo]['posX'] + posYcand = allhalos.loc[idhalo]['posY'] + posZcand = allhalos.loc[idhalo]['posZ'] ext = "/512Parent/outputs/snapdir_063/snap_063" header=rsHD.snapshot_header(self.main.headertab.parentsimpath+ext) - print "------------------------------------------------" - print "Rockstar ID inside parent simulation: ",idhalo - print "------------------------------------------------" - print " Index:",nhalo - print " x-pos:",'{:.2f}'.format(float(allhalos.ix[idhalo]['posX'])), " \ [Mpc/h]" - print " y-pos:",'{:.2f}'.format(float(allhalos.ix[idhalo]['posY'])), " \ [Mpc/h]" - print " z-pos:",'{:.2f}'.format(float(allhalos.ix[idhalo]['posZ'])), " \ [Mpc/h]" - print " virial mass:",'{0:.2e}'.format(float(allhalos.ix[idhalo]['mvir'])/header.hubble),"\ [Msol]" - print " virial radius:",'{:.2f}'.format(float(allhalos.ix[idhalo]['rvir']))," \ [kpc]" - print "------------------------------------------------" + print("------------------------------------------------") + print("Rockstar ID inside parent simulation: ",idhalo) + print("------------------------------------------------") + print(" Index:",nhalo) + print(" x-pos:",'{:.2f}'.format(float(allhalos.loc[idhalo]['posX'])), " \\ [Mpc/h]") + print(" y-pos:",'{:.2f}'.format(float(allhalos.loc[idhalo]['posY'])), " \\ [Mpc/h]") + print(" z-pos:",'{:.2f}'.format(float(allhalos.loc[idhalo]['posZ'])), " \\ [Mpc/h]") + print(" virial mass:",'{0:.2e}'.format(float(allhalos.loc[idhalo]['mvir'])/header.hubble),"\\ [Msol]") + print(" virial radius:",'{:.2f}'.format(float(allhalos.loc[idhalo]['rvir']))," \\ [kpc]") + print("------------------------------------------------") self.rockstarhaloid = idhalo - self.haloposx = '{:.2f}'.format(float(allhalos.ix[idhalo]['posX'])) - self.haloposy = '{:.2f}'.format(float(allhalos.ix[idhalo]['posY'])) - self.haloposz = '{:.2f}'.format(float(allhalos.ix[idhalo]['posZ'])) - self.halomvir = '{0:.2e}'.format(float(allhalos.ix[idhalo]['mvir'])/header.hubble) - self.halorvir = '{:.2f}'.format(float(allhalos.ix[idhalo]['rvir'])) + self.haloposx = '{:.2f}'.format(float(allhalos.loc[idhalo]['posX'])) + self.haloposy = '{:.2f}'.format(float(allhalos.loc[idhalo]['posY'])) + self.haloposz = '{:.2f}'.format(float(allhalos.loc[idhalo]['posZ'])) + self.halomvir = '{0:.2e}'.format(float(allhalos.loc[idhalo]['mvir'])/header.hubble) + self.halorvir = '{:.2f}'.format(float(allhalos.loc[idhalo]['rvir'])) for Nrvir in self.nrvir: - print 'Constructing: H' + str(self.haloidselect) + 'NRVIR' + str(int(Nrvir)) + print('Constructing: H' + str(self.haloidselect) + 'NRVIR' + str(int(Nrvir))) Nrvir = float(Nrvir) ext = "/512Parent/outputs/snapdir_063/snap_063" snapPOS = rsHD.read_block(self.main.headertab.parentsimpath+ext,"POS ") - dx = allhalos.ix[idhalo]['posX'] - snapPOS[:,0] - dy = allhalos.ix[idhalo]['posY'] - snapPOS[:,1] - dz = allhalos.ix[idhalo]['posZ'] - snapPOS[:,2] + dx = allhalos.loc[idhalo]['posX'] - snapPOS[:,0] + dy = allhalos.loc[idhalo]['posY'] - snapPOS[:,1] + dz = allhalos.loc[idhalo]['posZ'] - snapPOS[:,2] R = np.sqrt(dx**2. + dy**2. + dz**2.) Rindex = np.where(R < Nrvir*rvircand/1000) currentpos = snapPOS[Rindex[0]] @@ -601,7 +601,7 @@ def _makeic_button_fired(self): self.extz=2.0*dz*1.12/header.boxsize if len(self.nrvir) == 1: - print "Visualising:", 'HALO' + str(self.haloidselect) + 'NRVIR' + str(int(self.nrvir[0])) + print("Visualising:", 'HALO' + str(self.haloidselect) + 'NRVIR' + str(int(self.nrvir[0]))) figure = self.main.display figure.clear() ax = figure.add_subplot(111) @@ -674,11 +674,11 @@ def _makeic_button_fired(self): filename = self.lagroutputdir + '/H' + str(self.haloidselect) + 'NRVIR' + str(int(Nrvir)) f2=open(filename,'w') - for iv in xrange(0,len(lagrPos[:,0])): + for iv in range(0,len(lagrPos[:,0])): f2.write(str(lagrPos[iv,0]/header.boxsize)+' '+str(lagrPos[iv,1]/header.boxsize)+' '+ str(lagrPos[iv,2]/header.boxsize)+'\n') f2.close() - print "Region constructed!" + print("Region constructed!") def __init__(self, main, **kwargs): diff --git a/install.py b/install.py index 7a7fb1a..6781046 100644 --- a/install.py +++ b/install.py @@ -47,7 +47,7 @@ def _makelibdir_button_fired(self): def _installfftw2_button_fired(self): tmplibpath = './lib/installs/fftw-2.1.5/' - print "Installing:",tmplibpath + print("Installing:",tmplibpath) flagstr = '--enable-type-prefix --enable-mpi' tmpdir = os.getcwd() @@ -89,11 +89,11 @@ def _installfftw2_button_fired(self): os.system("make clean") os.chdir(tmpdir) - print "FFTW2 INSTALLED:",str(self.fftw2dir) + print("FFTW2 INSTALLED:",str(self.fftw2dir)) def _installfftw3_button_fired(self): tmplibpath = './lib/installs/fftw-3.3.3/' - print "Installing:",tmplibpath + print("Installing:",tmplibpath) flagstr = '--enable-threads --enable-float' tmpdir = os.getcwd() @@ -135,11 +135,11 @@ def _installfftw3_button_fired(self): os.system("make clean") os.chdir(tmpdir) - print "FFTW3 INSTALLED:",str(self.fftw3dir) + print("FFTW3 INSTALLED:",str(self.fftw3dir)) def _installhdf5_button_fired(self): tmplibpath = './lib/installs/hdf5-1.8.10/' - print "Installing:",tmplibpath + print("Installing:",tmplibpath) #(/home/bgriffen/lib/szip,/home/bgriffen/lib/zlib/include,/home/bgriffen/lib/zlib/lib) #flagstr = '--enable-fortran --enable-cxx --with-szlib=%s --with-zlib=%s,%s' flagstr = '--enable-fortran --enable-cxx' @@ -151,11 +151,11 @@ def _installhdf5_button_fired(self): os.system("make install") os.system("make clean") os.chdir(tmpdir) - print "HDF5 INSTALLED:",str(self.hdf5dir) + print("HDF5 INSTALLED:",str(self.hdf5dir)) def _installgsl_button_fired(self): tmplibpath = './lib/installs/gsl-1.9/' - print "Installing:",tmplibpath + print("Installing:",tmplibpath) tmpdir = os.getcwd() os.chdir(tmplibpath) os.system("make clean") @@ -164,11 +164,11 @@ def _installgsl_button_fired(self): os.system("make install") os.system("make clean") os.chdir(tmpdir) - print "GSL INSTALLED:",str(self.gsldir) + print("GSL INSTALLED:",str(self.gsldir)) def _installszip_button_fired(self): tmplibpath = './lib/installs/szip-2.1/' - print "Installing:",tmplibpath + print("Installing:",tmplibpath) tmpdir = os.getcwd() os.chdir(tmplibpath) os.system("make clean") @@ -178,11 +178,11 @@ def _installszip_button_fired(self): os.system("make install") os.system("make clean") os.chdir(tmpdir) - print "SZIP INSTALLED:",str(self.szipdir) + print("SZIP INSTALLED:",str(self.szipdir)) def _installzlib_button_fired(self): tmplibpath = './lib/installs/zlib-1.2.8/' - print "Installing:",tmplibpath + print("Installing:",tmplibpath) tmpdir = os.getcwd() os.chdir(tmplibpath) os.system("make clean") @@ -191,7 +191,7 @@ def _installzlib_button_fired(self): os.system("make install") os.system("make clean") os.chdir(tmpdir) - print "ZLIB INSTALLED:",str(self.zlibdir) + print("ZLIB INSTALLED:",str(self.zlibdir)) def __init__(self, main, **kwargs): HasTraits.__init__(self) @@ -202,4 +202,4 @@ def __init__(self, main, **kwargs): self.hdf5dir = self.libpath + '/hdf5' self.gsldir = self.libpath + '/gsl' self.zlibdir = self.libpath + '/zlib' - \ No newline at end of file + From 769e6ef67e16680c38e978fc2ca14742f0396b98 Mon Sep 17 00:00:00 2001 From: Brendan Griffen Date: Mon, 27 Jul 2026 21:49:39 +1000 Subject: [PATCH 05/19] refactor: migrate off the retired enthought.* Traits namespace The `enthought.*` namespace packages were dropped in ETS 4 (2011), so `enthought.traits.api` and friends do not exist in any installable release. Map them onto their modern homes: enthought.traits.api -> traits.api enthought.traits.ui.api -> traitsui.api enthought.traits.ui.wx.editor -> traitsui.wx.editor enthought.traits.ui.wx.basic_editor_factory-> traitsui.basic_editor_factory enthought.enable.api -> enable.api wxPython remains the toolkit, so this is an import-path change only. Also replace the four star imports with explicit names and declare `__all__`, since `from Common import *` in every tab previously re-exported the entire transitive import graph. Two concrete problems this fixes: - ics.py uses `patches.Rectangle` without importing it. That resolved only because `from matplotlib import *` happened to run after pyplot had been imported, which populates `matplotlib.patches` as a side effect. Reordering two lines in Common.py would have broken the ICs tab. It is now imported explicitly. - `Figure` now comes from `matplotlib.figure` rather than `matplotlib.pyplot`, so the module no longer pulls in pyplot's global figure registry, which should not be involved in an embedded canvas. Dropped 11 names that were imported but never used (TableEditor, EnumEditor, RangeEditor, Handler, Label, NoButtons, Spring, spring, UItem, TabularEditor, TabularAdapter) plus unused sys, socket, pylab and randint imports. Verified by AST analysis that every free name in all nine tab modules is still supplied; the only unresolved name is the pre-existing dead `Analysis` reference in main.py, removed in a later commit. --- Common.py | 159 ++++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 120 insertions(+), 39 deletions(-) diff --git a/Common.py b/Common.py index 15010f0..b82086e 100644 --- a/Common.py +++ b/Common.py @@ -1,46 +1,130 @@ -from matplotlib.pyplot import Figure -from matplotlib.backends.backend_wxagg import FigureCanvasWxAgg as FigureCanvas -from matplotlib.backends.backend_wx import NavigationToolbar2Wx +"""Shared imports and the matplotlib-in-TraitsUI figure editor. -from enthought.traits.api import * -from enthought.traits.ui.api import View,UItem, Item,Group, Heading, Label, \ - HSplit, Handler, CheckListEditor, EnumEditor, TableEditor, \ - ListEditor, Tabbed, VGroup, HGroup, RangeEditor, Spring, spring -from enthought.traits.ui.menu import NoButtons -from traitsui.ui_editors.array_view_editor import ArrayViewEditor -#from traits.api import Array -#from enthought.traits.api import Any, Instance -from enthought.traits.ui.wx.editor import Editor -from enthought.traits.ui.wx.basic_editor_factory import BasicEditorFactory -import numpy as np -from enthought.enable.api import ColorTrait -from traitsui.api import TabularEditor -from traitsui.tabular_adapter import TabularAdapter -from matplotlib import * +Every tab does ``from Common import *``, so this module defines ``__all__`` +to make that surface explicit rather than leaking the whole transitive +import graph. +""" import os -import sys -import wx import platform -import socket -import random import subprocess -import pylab as plt +import numpy as np +import wx +from matplotlib import patches +from matplotlib.backends.backend_wx import NavigationToolbar2Wx +from matplotlib.backends.backend_wxagg import FigureCanvasWxAgg as FigureCanvas +from matplotlib.figure import Figure + +from enable.api import ColorTrait +from traits.api import ( + Array, + Bool, + Button, + Directory, + Enum, + Float, + HasTraits, + Instance, + Int, + List, + Range, + Str, +) +from traitsui.api import ( + CheckListEditor, + Group, + Heading, + HGroup, + HSplit, + Item, + ListEditor, + Tabbed, + VGroup, + View, +) +from traitsui.basic_editor_factory import BasicEditorFactory +from traitsui.ui_editors.array_view_editor import ArrayViewEditor +from traitsui.wx.editor import Editor -import modules.readsnapshots.readsnapHDF5 as rsHD -import modules.readsnapshots.readsnap as rs -import modules.readsnapshots.readids as readids -import modules.readhalos.readsubf as readsubf import modules.readhalos.RSDataReaderv2 as RSDataReader -from modules.brendanlib.grifflib import * +import modules.readhalos.readsubf as readsubf +import modules.readsnapshots.readsnap as rs +import modules.readsnapshots.readsnapHDF5 as rsHD +from modules.brendanlib.grifflib import ( + COM, + CorrectPos, + addsubtitle, + cosmoconstant, + drawcircle, + getcandidatelist, + getcentext, + getlagrxyz, + placenormtext, +) -from random import randint +__all__ = [ + # Standard library and numerics + "os", + "platform", + "subprocess", + "np", + "wx", + # Matplotlib + "Figure", + "FigureCanvas", + "NavigationToolbar2Wx", + "patches", + # Traits + "Array", + "Bool", + "Button", + "ColorTrait", + "Directory", + "Enum", + "Float", + "HasTraits", + "Instance", + "Int", + "List", + "Range", + "Str", + # TraitsUI + "ArrayViewEditor", + "BasicEditorFactory", + "CheckListEditor", + "Editor", + "Group", + "HGroup", + "HSplit", + "Heading", + "Item", + "ListEditor", + "Tabbed", + "VGroup", + "View", + "MPLFigureEditor", + # Simulation I/O (external `modules` package) + "RSDataReader", + "readsubf", + "rs", + "rsHD", + "COM", + "CorrectPos", + "addsubtitle", + "cosmoconstant", + "drawcircle", + "getcandidatelist", + "getcentext", + "getlagrxyz", + "placenormtext", +] class _MPLFigureEditor(Editor): + """Embeds a matplotlib canvas, with toolbar, in a TraitsUI view.""" - scrollable = True + scrollable = True def init(self, parent): self.control = self._create_canvas(parent) @@ -50,22 +134,19 @@ def update_editor(self): pass def _create_canvas(self, parent): - """ Create the MPL canvas. """ - # The panel lets us add additional controls. - + """Create the MPL canvas inside a panel that can hold the toolbar.""" panel = wx.Panel(parent, -1, style=wx.CLIP_CHILDREN) sizer = wx.BoxSizer(wx.VERTICAL) panel.SetSizer(sizer) - - # matplotlib commands to create a canvas + mpl_control = FigureCanvas(panel, -1, self.value) sizer.Add(mpl_control, 1, wx.LEFT | wx.TOP | wx.GROW) toolbar = NavigationToolbar2Wx(mpl_control) sizer.Add(toolbar, 0, wx.EXPAND) - self.value.canvas.SetMinSize((10,10)) - + self.value.canvas.SetMinSize((10, 10)) + return panel -class MPLFigureEditor(BasicEditorFactory): - klass = _MPLFigureEditor \ No newline at end of file +class MPLFigureEditor(BasicEditorFactory): + klass = _MPLFigureEditor From 764de4aa74f2ee1a3fb8546f4c6ee6db7b6b29a4 Mon Sep 17 00:00:00 2001 From: Brendan Griffen Date: Mon, 27 Jul 2026 21:52:22 +1000 Subject: [PATCH 06/19] refactor: move modules into a src/cme package The eleven modules sat flat in the repository root, importable only by running from that directory, with no packaging metadata. Move them into a src layout and give the tabs names that describe what they are: main.py -> src/cme/app.py Common.py -> src/cme/ui/common.py header.py -> src/cme/ui/home.py (it is the Home tab) gadgetrun.py -> src/cme/ui/gadget.py contam.py -> src/cme/ui/contamination.py ics.py, halos.py, candidates.py, install.py, mergertree.py -> src/cme/ui/ This commit is moves plus the import rewrites they force, so the diff is reviewable as a rename. `src/cme/ui/` is where the remaining display-coupled code lives; subsequent commits pull the computation and file-format writing out into sibling packages that can be tested without a display. Also add pyproject.toml, since the dependency set previously had to be reverse-engineered from imports. Note the split: the base dependency set is numpy and pandas only, because the extracted modules that follow are display-free and must stay testable in CI without a GUI toolkit. Matplotlib, traits, traitsui, enable, mayavi and wxPython move to a `gui` extra. Adds a `cme` console script entry point, replacing `python main.py`. --- pyproject.toml | 70 +++++++++++++++++++++++ __init__.py => src/cme/__init__.py | 2 +- main.py => src/cme/app.py | 35 +++++++----- src/cme/ui/__init__.py | 7 +++ candidates.py => src/cme/ui/candidates.py | 2 +- Common.py => src/cme/ui/common.py | 0 contam.py => src/cme/ui/contamination.py | 2 +- gadgetrun.py => src/cme/ui/gadget.py | 2 +- halos.py => src/cme/ui/halos.py | 2 +- header.py => src/cme/ui/home.py | 2 +- ics.py => src/cme/ui/ics.py | 2 +- install.py => src/cme/ui/install.py | 2 +- mergertree.py => src/cme/ui/mergertree.py | 2 +- 13 files changed, 106 insertions(+), 24 deletions(-) create mode 100644 pyproject.toml rename __init__.py => src/cme/__init__.py (87%) rename main.py => src/cme/app.py (89%) create mode 100644 src/cme/ui/__init__.py rename candidates.py => src/cme/ui/candidates.py (99%) rename Common.py => src/cme/ui/common.py (100%) rename contam.py => src/cme/ui/contamination.py (99%) rename gadgetrun.py => src/cme/ui/gadget.py (99%) rename halos.py => src/cme/ui/halos.py (99%) rename header.py => src/cme/ui/home.py (99%) rename ics.py => src/cme/ui/ics.py (99%) rename install.py => src/cme/ui/install.py (99%) rename mergertree.py => src/cme/ui/mergertree.py (99%) diff --git a/pyproject.toml b/pyproject.toml new file mode 100644 index 0000000..8dce8ab --- /dev/null +++ b/pyproject.toml @@ -0,0 +1,70 @@ +[build-system] +requires = ["setuptools>=68", "wheel"] +build-backend = "setuptools.build_meta" + +[project] +name = "cme" +version = "0.1.0" +description = "Caterpillar Made Easy: a GUI for setting up, running and inspecting cosmological zoom-in simulations" +readme = "README.md" +requires-python = ">=3.9" +license = { file = "LICENSE" } +authors = [{ name = "Brendan Griffen", email = "brendan.f.griffen@gmail.com" }] +keywords = ["cosmology", "simulation", "zoom-in", "gadget", "music", "rockstar"] +classifiers = [ + "Development Status :: 3 - Alpha", + "Intended Audience :: Science/Research", + "License :: OSI Approved :: MIT License", + "Programming Language :: Python :: 3", + "Topic :: Scientific/Engineering :: Astronomy", +] + +# Only what the display-free modules need. The GUI needs the `gui` extra, and +# the simulation I/O readers (the `modules` package) are not distributed here -- +# see README.md. +dependencies = [ + "numpy>=1.21", + "pandas>=1.3", +] + +[project.optional-dependencies] +gui = [ + "matplotlib>=3.5", + "traits>=6.3", + "traitsui>=7.3", + "enable>=5.3", + "mayavi>=4.8", + "wxPython>=4.2", +] +dev = [ + "pytest>=7.0", + "ruff>=0.5", +] + +[project.urls] +Homepage = "https://github.com/bgriffen/cme" +Issues = "https://github.com/bgriffen/cme/issues" + +[project.scripts] +cme = "cme.app:main" + +[tool.setuptools.packages.find] +where = ["src"] + +[tool.pytest.ini_options] +testpaths = ["tests"] +addopts = "-q" + +[tool.ruff] +line-length = 100 +src = ["src", "tests"] + +[tool.ruff.lint] +select = ["E", "F", "W", "I", "UP", "B", "C4", "SIM"] +# The GUI modules are wide TraitsUI view declarations where a nested call +# chain reads better than a wrapped one; they are excluded from line-length +# and star-import rules until they are broken up. +ignore = ["E501"] + +[tool.ruff.lint.per-file-ignores] +"src/cme/ui/*.py" = ["F403", "F405"] diff --git a/__init__.py b/src/cme/__init__.py similarity index 87% rename from __init__.py rename to src/cme/__init__.py index ee4d993..c1c26ed 100644 --- a/__init__.py +++ b/src/cme/__init__.py @@ -3,4 +3,4 @@ """ A tool for constructing and analysing cosmological simulations. """ __author__ = "Brendan Griffen " -__version__ = "0.0.1" +__version__ = "0.1.0" diff --git a/main.py b/src/cme/app.py similarity index 89% rename from main.py rename to src/cme/app.py index 0f04bba..1716e8d 100644 --- a/main.py +++ b/src/cme/app.py @@ -1,19 +1,20 @@ +"""Application entry point: assembles the tabbed main window.""" -from header import Header -from install import InstallFiles -from ics import InitialConditions -from gadgetrun import GadgetRun -from halos import HaloFind -from mergertree import MergerTree -#from analysis import Analysis -from contam import Contamination -from candidates import Candidates - -from tvtk.pyface.scene_editor import SceneEditor -from mayavi.tools.mlab_scene_model import MlabSceneModel from mayavi.core.ui.mayavi_scene import MayaviScene +from mayavi.tools.mlab_scene_model import MlabSceneModel +from tvtk.pyface.scene_editor import SceneEditor + +from cme.ui.candidates import Candidates +from cme.ui.common import * +from cme.ui.contamination import Contamination +from cme.ui.gadget import GadgetRun +from cme.ui.halos import HaloFind +from cme.ui.home import Header +from cme.ui.ics import InitialConditions +from cme.ui.install import InstallFiles +from cme.ui.mergertree import MergerTree -from Common import * +#from analysis import Analysis class ApplicationMain(HasTraits): @@ -128,6 +129,10 @@ def __init__(self, **kwargs): self.markersize = 2 self.markerstyle = 'o' +def main(): + """Launch the application. Wired up as the ``cme`` console script.""" + ApplicationMain().configure_traits() + + if __name__ == '__main__': - app = ApplicationMain() - app.configure_traits() + main() diff --git a/src/cme/ui/__init__.py b/src/cme/ui/__init__.py new file mode 100644 index 0000000..dcceb22 --- /dev/null +++ b/src/cme/ui/__init__.py @@ -0,0 +1,7 @@ +"""TraitsUI tabs. + +Each module here defines one ``HasTraits`` tab of the main window. Tabs are +intentionally thin: they bind widgets to state and delegate the actual work to +the display-free modules in :mod:`cme.naming`, :mod:`cme.cosmology`, +:mod:`cme.science` and :mod:`cme.writers`. +""" diff --git a/candidates.py b/src/cme/ui/candidates.py similarity index 99% rename from candidates.py rename to src/cme/ui/candidates.py index 7b1771c..458d153 100644 --- a/candidates.py +++ b/src/cme/ui/candidates.py @@ -1,4 +1,4 @@ -from Common import * +from cme.ui.common import * from numpy.random import random class Candidates(HasTraits): diff --git a/Common.py b/src/cme/ui/common.py similarity index 100% rename from Common.py rename to src/cme/ui/common.py diff --git a/contam.py b/src/cme/ui/contamination.py similarity index 99% rename from contam.py rename to src/cme/ui/contamination.py index b82a823..916b41f 100644 --- a/contam.py +++ b/src/cme/ui/contamination.py @@ -1,4 +1,4 @@ -from Common import * +from cme.ui.common import * import modules.mergertrees.MTCatalogue as MT class Contamination(HasTraits): diff --git a/gadgetrun.py b/src/cme/ui/gadget.py similarity index 99% rename from gadgetrun.py rename to src/cme/ui/gadget.py index a88bbe6..3a0973a 100644 --- a/gadgetrun.py +++ b/src/cme/ui/gadget.py @@ -1,4 +1,4 @@ -from Common import * +from cme.ui.common import * import modules.mergertrees.MTCatalogue as MT class GadgetRun(HasTraits): diff --git a/halos.py b/src/cme/ui/halos.py similarity index 99% rename from halos.py rename to src/cme/ui/halos.py index a9a5164..aabe0e8 100644 --- a/halos.py +++ b/src/cme/ui/halos.py @@ -1,4 +1,4 @@ -from Common import * +from cme.ui.common import * class HaloFind(HasTraits): diff --git a/header.py b/src/cme/ui/home.py similarity index 99% rename from header.py rename to src/cme/ui/home.py index c7ea88d..e68758c 100644 --- a/header.py +++ b/src/cme/ui/home.py @@ -1,4 +1,4 @@ -from Common import * +from cme.ui.common import * import getpass from os.path import expanduser class Header(HasTraits): diff --git a/ics.py b/src/cme/ui/ics.py similarity index 99% rename from ics.py rename to src/cme/ui/ics.py index be09510..10f9ca5 100644 --- a/ics.py +++ b/src/cme/ui/ics.py @@ -1,4 +1,4 @@ -from Common import * +from cme.ui.common import * from glob import glob import random diff --git a/install.py b/src/cme/ui/install.py similarity index 99% rename from install.py rename to src/cme/ui/install.py index 6781046..6296e0d 100644 --- a/install.py +++ b/src/cme/ui/install.py @@ -1,4 +1,4 @@ -from Common import * +from cme.ui.common import * class InstallFiles(HasTraits): diff --git a/mergertree.py b/src/cme/ui/mergertree.py similarity index 99% rename from mergertree.py rename to src/cme/ui/mergertree.py index 1e4aee3..038c068 100644 --- a/mergertree.py +++ b/src/cme/ui/mergertree.py @@ -1,4 +1,4 @@ -from Common import * +from cme.ui.common import * import modules.mergertrees.MTCatalogue as MT class MergerTree(HasTraits): From 1e3f50a33648440c2374dd39e22aea37abf0ee1d Mon Sep 17 00:00:00 2001 From: Brendan Griffen Date: Mon, 27 Jul 2026 22:01:25 +1000 Subject: [PATCH 07/19] refactor: extract the path conventions into cme.naming The run-directory convention H_B_Z_P_LN_LX_O_NV was rebuilt inline in 18 places, and the Lagrangian region paths in 7 more. Nothing recorded the convention in one place, so the copies had drifted. Replace them all with cme/naming.py, which is display-free and unit tested. Three bugs were a direct consequence of the duplication, and are fixed by routing every caller through the shared helpers: 1. Snapshot indices were zero-padded by concatenation: "groups_0" + str(n). That is only correct for two-digit snapshot numbers. Snapshot 5 produced "groups_05" where Gadget writes "groups_005", so every existence check against a single-digit snapshot reported "not found"; the zoom tabs default to snapshot 255, which produced "snapdir_0255". Now formatted to a fixed width of 3. 2. The contamination and halo tabs hardcoded "_Z127" rather than reading their own zinit trait, so any run generated at a different starting redshift was looked up in a directory the ICs tab had never created. zinit is now a parameter that defaults to 127. 3. Base paths were concatenated as `self.gadpath + 'halos/H'`, which silently produced ".../datahalos/H190897" whenever the path trait had no trailing slash. Now joined with os.path.join. Also replaces the hardcoded snapshot 63 in the Gadget existence check with the last snapshot implied by the requested output count, so the table is correct for output counts other than 64. The contamination tab shrinks by 50 lines: eight of its parameter handlers were byte-identical 8-line blocks, now one `_refresh()` call each. That collapse also fixes AUDIT 2.1.3, where four of those handlers wrote `self.makeactive == True` instead of `=`, leaving the plot buttons permanently disabled after changing overlap or snapshot number. Two further defects were identified while extracting this and are fixed in later commits, to keep this one mechanical: the ICs tab iterates over its `lmin` Enum with `for lmini in self.lmin`, which iterates the characters of the string and so breaks for levels >= 10; and the Lagrangian display traits format a List with str(), producing "NRVIR['1']". 23 tests, covering the format, the round-trip parse, the type coercion the GUI forces (strings from CheckListEditor, ints from Range), and a regression test per bug above. --- src/cme/naming.py | 188 ++++++++++++++++++++++++++++++++++++ src/cme/ui/contamination.py | 177 ++++++++++++--------------------- src/cme/ui/gadget.py | 55 ++++------- src/cme/ui/halos.py | 35 +++---- src/cme/ui/ics.py | 35 +++---- tests/test_naming.py | 149 ++++++++++++++++++++++++++++ 6 files changed, 455 insertions(+), 184 deletions(-) create mode 100644 src/cme/naming.py create mode 100644 tests/test_naming.py diff --git a/src/cme/naming.py b/src/cme/naming.py new file mode 100644 index 0000000..596bc81 --- /dev/null +++ b/src/cme/naming.py @@ -0,0 +1,188 @@ +"""The run-directory and Lagrangian-region path conventions. + +A resimulation is identified by its full parameter set, encoded into a +directory name:: + + H_B_Z_P_LN_LX_O_NV + +for example ``H190897_BE_Z127_P7_LN7_LX11_O4_NV4``. + +This convention was previously rebuilt inline in 18 places across the GUI +tabs, which is how the two inconsistencies fixed here arose: the +contamination and halo tabs hardcoded ``_Z127`` rather than reading their own +``zinit``, and Gadget snapshot indices were zero-padded by string +concatenation rather than to a fixed width. Both are covered by tests. +""" + +import os +import re + +#: Gadget zero-pads snapshot and group directory indices to three digits. +GADGET_INDEX_WIDTH = 3 + +#: The starting redshift used throughout the Caterpillar suite. +DEFAULT_ZINIT = 127 + +#: Single-letter codes for the MUSIC refinement region shape. +REGION_CODES = {"box": "B", "ellipsoid": "E"} + +_RUN_DIR_RE = re.compile( + r"^H(?P\d+)" + r"_B(?P[BE])" + r"_Z(?P\d+)" + r"_P(?P\d+)" + r"_LN(?P\d+)" + r"_LX(?P\d+)" + r"_O(?P\d+)" + r"_NV(?P\d+)$" +) + + +def region_code(region): + """Return the single-letter code for a refinement region shape. + + >>> region_code("ellipsoid") + 'E' + >>> region_code("box") + 'B' + """ + try: + return REGION_CODES[str(region).lower()] + except KeyError: + raise ValueError( + f"unknown region {region!r}; expected one of {sorted(REGION_CODES)}" + ) from None + + +def run_dir_name( + halo_id, + region, + padding, + levelmin, + levelmax, + overlap, + nrvir, + zinit=DEFAULT_ZINIT, +): + """Build the directory name encoding a resimulation's parameters. + + Numeric arguments are coerced with ``int``, because the GUI supplies them + either as integer ``Range`` traits or as strings from a ``CheckListEditor`` + depending on the tab. + + >>> run_dir_name(190897, "ellipsoid", 7, 7, 11, 4, 4) + 'H190897_BE_Z127_P7_LN7_LX11_O4_NV4' + """ + return ( + f"H{int(halo_id)}" + f"_B{region_code(region)}" + f"_Z{int(zinit)}" + f"_P{int(padding)}" + f"_LN{int(levelmin)}" + f"_LX{int(levelmax)}" + f"_O{int(overlap)}" + f"_NV{int(nrvir)}" + ) + + +def parse_run_dir_name(name): + """Inverse of :func:`run_dir_name`; returns a dict, or ``None`` if no match. + + >>> parse_run_dir_name("H190897_BE_Z127_P7_LN7_LX11_O4_NV4")["levelmax"] + 11 + """ + match = _RUN_DIR_RE.match(str(name)) + if match is None: + return None + parts = match.groupdict() + code = parts.pop("region_code") + parsed = {key: int(value) for key, value in parts.items()} + parsed["region"] = next(k for k, v in REGION_CODES.items() if v == code) + return parsed + + +def halo_dir(base, halo_id): + """Directory holding every resimulation of one halo.""" + return os.path.join(str(base), "halos", f"H{int(halo_id)}") + + +def run_dir(base, halo_id, **kwargs): + """Full path to one resimulation's directory. + + ``kwargs`` are forwarded to :func:`run_dir_name`. + """ + return os.path.join( + halo_dir(base, halo_id), run_dir_name(halo_id=halo_id, **kwargs) + ) + + +def outputs_dir(base, halo_id, **kwargs): + """Directory Gadget writes snapshots and group catalogues into.""" + return os.path.join(run_dir(base, halo_id, **kwargs), "outputs") + + +def job_name(halo_id, region, padding, levelmax, nrvir): + """Short scheduler job name for a resimulation. + + The halo ID is truncated to two digits to keep the name inside queue name + limits, matching the existing convention. + + >>> job_name(190897, "ellipsoid", 7, 11, 4) + 'H19EP7L11N4' + """ + return ( + f"H{str(int(halo_id))[:2]}" + f"{region_code(region)}" + f"P{int(padding)}" + f"L{int(levelmax)}" + f"N{int(nrvir)}" + ) + + +def rockstar_dir(base, halo_id, **kwargs): + """Directory holding one resimulation's Rockstar catalogue.""" + return os.path.join(run_dir(base, halo_id, **kwargs), "rockstardata") + + +def snapshot_index(snapnum): + """Zero-pad a snapshot number to Gadget's fixed width. + + The previous inline form was ``"snapdir_0" + str(snapnum)``, which is only + correct for two-digit snapshot numbers -- it produced ``snapdir_05`` + instead of ``snapdir_005``, and ``snapdir_0255`` instead of ``snapdir_255``. + + >>> snapshot_index(63), snapshot_index(5), snapshot_index(255) + ('063', '005', '255') + """ + return f"{int(snapnum):0{GADGET_INDEX_WIDTH}d}" + + +def snapshot_dir(outputs, snapnum): + """Path to one snapshot's directory inside an ``outputs`` directory.""" + return os.path.join(str(outputs), f"snapdir_{snapshot_index(snapnum)}") + + +def snapshot_base(outputs, snapnum): + """Path stem the Gadget readers expect for a snapshot (no file suffix).""" + return os.path.join( + snapshot_dir(outputs, snapnum), f"snap_{snapshot_index(snapnum)}" + ) + + +def groups_dir(outputs, snapnum): + """Path to one snapshot's FOF/SUBFIND group catalogue directory.""" + return os.path.join(str(outputs), f"groups_{snapshot_index(snapnum)}") + + +def lagr_region_path(lagr_dir, halo_id, nrvir): + """Path to the MUSIC region point file for one halo and ``nrvir``. + + >>> lagr_region_path("/data/ics/lagr", 190897, 4) + '/data/ics/lagr/H190897NRVIR4' + """ + return os.path.join(str(lagr_dir), f"H{int(halo_id)}NRVIR{int(nrvir)}") + + +def lagr_header_path(lagr_dir, halo_id, nrvir): + """Path to the ``.head`` file holding a region's centre and extent.""" + return lagr_region_path(lagr_dir, halo_id, nrvir) + ".head" diff --git a/src/cme/ui/contamination.py b/src/cme/ui/contamination.py index 916b41f..ce9b7ee 100644 --- a/src/cme/ui/contamination.py +++ b/src/cme/ui/contamination.py @@ -1,6 +1,9 @@ from cme.ui.common import * import modules.mergertrees.MTCatalogue as MT +from cme import naming + + class Contamination(HasTraits): gadpath = Directory @@ -79,22 +82,12 @@ class Contamination(HasTraits): Group(Item(name='haloidlist',show_label=False,style='readonly',editor=ListEditor(style='readonly',columns=6,rows=4)),label='Halo Sample',show_border=True)) def _checkcontam_button_fired(self): - self.foldername = 'H' + str(self.haloid) + \ - '_B' + str(self.boxtype.upper())[0] + \ - '_Z127' + \ - '_P' + str(self.padding) + \ - '_LN' + str(self.lmin) + \ - '_LX' + str(self.lmax) + \ - '_O' + str(self.overlap) + \ - '_NV' + str(self.nrvir) - - dircheck = self.gadpath + 'halos/H' + str(self.haloid) + '/' + self.foldername + "/outputs" \ - + "/groups_0" + str(self.snapnum) + self.foldername = naming.run_dir_name(halo_id=self.haloid, **self._run_kwargs()) + hubble = 0.6711 - if os.path.exists(dircheck): - - tmppath = self.gadpath + 'halos/H' + str(self.haloid) + '/' + self.foldername + "/outputs" - filepath = tmppath + "/snapdir_0" + str(self.snapnum) + "/snap_0" + str(self.snapnum) + if os.path.exists(self.groups_dir()): + + tmppath = self.outputs_dir() s = readsubf.subfind_catalog(tmppath, self.snapnum) #ids = readsubf.subf_ids(tmppath, self.snapnum, 0, 0, read_all=1) mgroup = s.group_m_mean200*10**10/hubble @@ -203,85 +196,61 @@ def _checkcontam_button_fired(self): else: self.datastatus = "Data not found, not plotted." + def _run_kwargs(self): + """The parameter set identifying the resimulation currently selected.""" + return dict(region=self.boxtype, + padding=self.padding, + levelmin=self.lmin, + levelmax=self.lmax, + overlap=self.overlap, + nrvir=self.nrvir, + zinit=self.zinit) + + def outputs_dir(self): + """Directory Gadget wrote this run's snapshots and catalogues into.""" + return naming.outputs_dir(self.gadpath, self.haloid, **self._run_kwargs()) + + def groups_dir(self): + """Directory holding the FOF/SUBFIND catalogue for the chosen snapshot.""" + return naming.groups_dir(self.outputs_dir(), self.snapnum) + + def _refresh(self, replot=False): + """Recompute the run name and whether its group catalogue exists. + + Every parameter change funnels through here, so the displayed folder + name and the enabled state of the plot buttons cannot drift apart. + """ + self.foldername = naming.run_dir_name(halo_id=self.haloid, **self._run_kwargs()) + self.makeactive = os.path.exists(self.groups_dir()) + if replot and self.makeactive: + self._checkcontam_button_fired() + def _haloid_changed(self): - #self.checkcontam_button_fired() - self.foldername = 'H' + str(self.haloid) + '_B' + str(self.boxtype.upper())[0] + '_Z127' + '_P' + str(self.padding) + '_LN' + str(self.lmin) + '_LX' + str(self.lmax) + '_O' + str(self.overlap) + '_NV' + str(self.nrvir) - dircheck = self.gadpath + 'halos/H' + str(self.haloid) + '/' + self.foldername + "/outputs/groups_0" + str(self.snapnum) - if os.path.exists(dircheck): - self.makeactive = True - else: - self.makeactive = False - + self._refresh() + def _boxtype_changed(self): - self.foldername = 'H' + str(self.haloid) + '_B' + str(self.boxtype.upper())[0] + '_Z127' + '_P' + str(self.padding) + '_LN' + str(self.lmin) + '_LX' + str(self.lmax) + '_O' + str(self.overlap) + '_NV' + str(self.nrvir) - dircheck = self.gadpath + 'halos/H' + str(self.haloid) + '/' + self.foldername + "/outputs/groups_0" + str(self.snapnum) - if os.path.exists(dircheck): - self.makeactive = True - else: - self.makeactive = False - + self._refresh() + def _nrvir_changed(self): - self.foldername = 'H' + str(self.haloid) + '_B' + str(self.boxtype.upper())[0] + '_Z127' + '_P' + str(self.padding) + '_LN' + str(self.lmin) + '_LX' + str(self.lmax) + '_O' + str(self.overlap) + '_NV' + str(self.nrvir) - dircheck = self.gadpath + 'halos/H' + str(self.haloid) + '/' + self.foldername + "/outputs/groups_0" + str(self.snapnum) - if os.path.exists(dircheck): - self._checkcontam_button_fired() - self.makeactive = True - else: - self.makeactive = False - + self._refresh(replot=True) + def _deltar_changed(self): - self.foldername = 'H' + str(self.haloid) + '_B' + str(self.boxtype.upper())[0] + '_Z127' + '_P' + str(self.padding) + '_LN' + str(self.lmin) + '_LX' + str(self.lmax) + '_O' + str(self.overlap) + '_NV' + str(self.nrvir) - dircheck = self.gadpath + 'halos/H' + str(self.haloid) + '/' + self.foldername + "/outputs/groups_0" + str(self.snapnum) - if os.path.exists(dircheck): - self.makeactive = True - self._checkcontam_button_fired() - else: - self.makeactive = False + self._refresh(replot=True) def _lmin_changed(self): - self.foldername = 'H' + str(self.haloid) + '_B' + str(self.boxtype.upper())[0] + '_Z127' + '_P' + str(self.padding) + '_LN' + str(self.lmin) + '_LX' + str(self.lmax) + '_O' + str(self.overlap) + '_NV' + str(self.nrvir) - dircheck = self.gadpath + 'halos/H' + str(self.haloid) + '/' + self.foldername + "/outputs/groups_0" + str(self.snapnum) - if os.path.exists(dircheck): - self.makeactive = True - self._checkcontam_button_fired() - else: - self.makeactive = False - + self._refresh(replot=True) + def _lmax_changed(self): - self.foldername = 'H' + str(self.haloid) + '_B' + str(self.boxtype.upper())[0] + '_Z127' + '_P' + str(self.padding) + '_LN' + str(self.lmin) + '_LX' + str(self.lmax) + '_O' + str(self.overlap) + '_NV' + str(self.nrvir) - dircheck = self.gadpath + 'halos/H' + str(self.haloid) + '/' + self.foldername + "/outputs/groups_0" + str(self.snapnum) - if os.path.exists(dircheck): - self.makeactive = True - self._checkcontam_button_fired() - else: - self.makeactive = False - + self._refresh(replot=True) + def _padding_changed(self): - self.foldername = 'H' + str(self.haloid) + '_B' + str(self.boxtype.upper())[0] + '_Z127' + '_P' + str(self.padding) + '_LN' + str(self.lmin) + '_LX' + str(self.lmax) + '_O' + str(self.overlap) + '_NV' + str(self.nrvir) - dircheck = self.gadpath + 'halos/H' + str(self.haloid) + '/' + self.foldername + "/outputs/groups_0" + str(self.snapnum) - if os.path.exists(dircheck): - self.makeactive = True - self._checkcontam_button_fired() - else: - self.makeactive = False - + self._refresh(replot=True) + def _overlap_changed(self): - self.foldername = 'H' + str(self.haloid) + '_B' + str(self.boxtype.upper())[0] + '_Z127' + '_P' + str(self.padding) + '_LN' + str(self.lmin) + '_LX' + str(self.lmax) + '_O' + str(self.overlap) + '_NV' + str(self.nrvir) - dircheck = self.gadpath + 'halos/H' + str(self.haloid) + '/' + self.foldername + "/outputs/groups_0" + str(self.snapnum) - if os.path.exists(dircheck): - self.makeactive == True - self._checkcontam_button_fired() - else: - self.makeactive == False - + self._refresh(replot=True) + def _snapnum_changed(self): - self.foldername = 'H' + str(self.haloid) + '_B' + str(self.boxtype.upper())[0] + '_Z127' + '_P' + str(self.padding) + '_LN' + str(self.lmin) + '_LX' + str(self.lmax) + '_O' + str(self.overlap) + '_NV' + str(self.nrvir) - dircheck = self.gadpath + 'halos/H' + str(self.haloid) + '/' + self.foldername + "/outputs/groups_0" + str(self.snapnum) - if os.path.exists(dircheck): - self.makeactive == True - self._checkcontam_button_fired() - else: - self.makeactive == False + self._refresh(replot=True) def _xvar_changed(self): if self.xvar == self.yvar: @@ -296,29 +265,18 @@ def _yvar_changed(self): self.xymatch = False def _plothalodist_button_fired(self): - self.foldername = 'H' + str(self.haloid) + \ - '_B' + str(self.boxtype.upper())[0] + \ - '_Z127' + \ - '_P' + str(self.padding) + \ - '_LN' + str(self.lmin) + \ - '_LX' + str(self.lmax) + \ - '_O' + str(self.overlap) + \ - '_NV' + str(self.nrvir) - + self.foldername = naming.run_dir_name(halo_id=self.haloid, **self._run_kwargs()) + figure = self.main.display figure.clear() ax = figure.add_subplot(111) ax = self.main.display.axes[0] - - dircheck = self.gadpath + 'halos/H' + str(self.haloid) + '/' + self.foldername + "/outputs" \ - + "/groups_0" + str(self.snapnum) - + hubble = 0.6711 - - if os.path.exists(dircheck): + + if os.path.exists(self.groups_dir()): self.datastatus = "Data exists, plotted." - tmppath = self.gadpath + 'halos/H' + str(self.haloid) + '/' + self.foldername + "/outputs" - filepath = tmppath + "/snapdir_0" + str(self.snapnum) + "/snap_0" + str(self.snapnum) + tmppath = self.outputs_dir() s = readsubf.subfind_catalog(tmppath, self.snapnum) #ids = readsubf.subf_ids(tmppath, self.snapnum, 0, 0, read_all=1) mgroup = s.group_m_mean200*10**10/hubble @@ -380,13 +338,11 @@ def _plothalodist_button_fired(self): def _plotmayavi_button_fired(self): self.main.scene.mlab.clf(figure=self.main.scene.mayavi_scene) - self.foldername = 'H' + str(self.haloid) + '_B' + str(self.boxtype.upper())[0] + '_Z127' + '_P' + str(self.padding) + '_LN' + str(self.lmin) + '_LX' + str(self.lmax) + '_O' + str(self.overlap) + '_NV' + str(self.nrvir) - dircheck = self.gadpath + 'halos/H' + str(self.haloid) + '/' + self.foldername + "/outputs/groups_0" + str(self.snapnum) + self.foldername = naming.run_dir_name(halo_id=self.haloid, **self._run_kwargs()) hubble = 0.6711 - if os.path.exists(dircheck): + if os.path.exists(self.groups_dir()): self.datastatus = "Data exists, plotted." - tmppath = self.gadpath + 'halos/H' + str(self.haloid) + '/' + self.foldername + "/outputs" - filepath = tmppath + "/snapdir_0" + str(self.snapnum) + "/snap_0" + str(self.snapnum) + tmppath = self.outputs_dir() s = readsubf.subfind_catalog(tmppath, self.snapnum) mgroup = s.group_m_mean200*10**10/hubble @@ -421,13 +377,8 @@ def __init__(self, main, **kwargs): self.haloidlist = self.main.candidatestab.haloid self.gadpath = self.main.headertab.datamasterpath - - self.foldername = 'H' + str(self.haloid) + '_B' + str(self.boxtype.upper())[0] + '_Z127' + '_P' + str(self.padding) + '_LN' + str(self.lmin) + '_LX' + str(self.lmax) + '_O' + str(self.overlap) + '_NV' + str(self.nrvir) - dircheck = self.gadpath + 'halos/H' + str(self.haloid) + '/' + self.foldername + "/outputs/groups_0" + str(self.snapnum) - if os.path.exists(dircheck): - self.makeactive = True - else: - self.makeactive = False + + self._refresh() diff --git a/src/cme/ui/gadget.py b/src/cme/ui/gadget.py index 3a0973a..220d38c 100644 --- a/src/cme/ui/gadget.py +++ b/src/cme/ui/gadget.py @@ -1,4 +1,6 @@ from cme.ui.common import * + +from cme import naming import modules.mergertrees.MTCatalogue as MT class GadgetRun(HasTraits): @@ -272,21 +274,14 @@ def _subscript_button_fired(self): for nrviri in self.nrvir: for paddingi in self.padding: for overlapi in self.overlap: - foldername = 'H' + str(self.haloid) + \ - '_B' + str(boxtypei.upper())[0] + \ - '_Z' + str(self.zinit) + \ - '_P' + str(paddingi) + \ - '_LN' + str(self.lmin) + \ - '_LX' + str(self.lmax) + \ - '_O' + str(overlapi) + \ - '_NV' + str(nrviri) - jobname = 'H' + str(self.haloid)[0:2] + \ - str(boxtypei.upper())[0] + \ - 'P' + str(paddingi) + \ - 'L' + str(self.lmax) + \ - 'N' + str(nrviri) - - filepath = self.gadpath + 'halos/H' + str(self.haloid) + '/' + foldername + run = dict(region=boxtypei, padding=paddingi, + levelmin=self.lmin, levelmax=self.lmax, + overlap=overlapi, nrvir=nrviri, zinit=self.zinit) + foldername = naming.run_dir_name(halo_id=self.haloid, **run) + jobname = naming.job_name(halo_id=self.haloid, region=boxtypei, + padding=paddingi, levelmax=self.lmax, + nrvir=nrviri) + filepath = naming.run_dir(self.gadpath, self.haloid, **run) if not os.path.exists(filepath): print("PATH NOT FOUND:",filepath) else: @@ -795,16 +790,11 @@ def _checkexistence_button_fired(self): for nrviri in self.nrvir: for paddingi in self.padding: for overlapi in self.overlap: - foldername = 'H' + str(self.haloid) + \ - '_B' + str(boxtypei.upper())[0] + \ - '_Z' + str(self.zinit) + \ - '_P' + str(paddingi) + \ - '_LN' + str(self.lmin) + \ - '_LX' + str(self.lmax) + \ - '_O' + str(overlapi) + \ - '_NV' + str(nrviri) - - filepath = self.gadpath + 'halos/H' + str(self.haloid) + '/' + foldername + run = dict(region=boxtypei, padding=paddingi, + levelmin=self.lmin, levelmax=self.lmax, + overlap=overlapi, nrvir=nrviri, zinit=self.zinit) + foldername = naming.run_dir_name(halo_id=self.haloid, **run) + filepath = naming.run_dir(self.gadpath, self.haloid, **run) #print filepath #print filepath + '/ics.0' @@ -815,15 +805,12 @@ def _checkexistence_button_fired(self): icfound = '+' except IOError: icfound = '-' - if os.path.exists(filepath + '/outputs/snapdir_063'): - gadfound = '+' - else: - gadfound = '-' - - if os.path.exists(filepath + '/outputs/groups_063'): - halosfound = '+' - else: - halosfound = '-' + outputs = os.path.join(filepath, 'outputs') + lastsnap = int(self.nintervals) - 1 + gadfound = '+' if os.path.exists( + naming.snapshot_dir(outputs, lastsnap)) else '-' + halosfound = '+' if os.path.exists( + naming.groups_dir(outputs, lastsnap)) else '-' txtdisplace += 0.03 placenormtext(ax,0.02, 0.98 - txtdisplace,str(self.haloid),10) diff --git a/src/cme/ui/halos.py b/src/cme/ui/halos.py index aabe0e8..489e8ce 100644 --- a/src/cme/ui/halos.py +++ b/src/cme/ui/halos.py @@ -1,5 +1,7 @@ from cme.ui.common import * +from cme import naming + class HaloFind(HasTraits): halopath = Directory @@ -127,6 +129,15 @@ def _parentorzoom_changed(self): if self.parentorzoom == 'zoom': self.snapshot = 255 + def _run_kwargs(self): + """The parameter set identifying the zoom resimulation being inspected.""" + return dict(region=self.boxtype, + padding=self.padding, + levelmin=self.lmin, + levelmax=self.lmax, + overlap=self.overlap, + nrvir=self.nrvir) + def gethostid(self): path = "/bigbang/data/AnnaGroup/caterpillar/halos" with open(path + "/halosummary.txt") as f: @@ -151,16 +162,8 @@ def gethalos_xy(self): self.datastatus = 'Parent not found.' elif self.parentorzoom == 'zoom': - self.foldername = 'H' + str(self.haloid) + \ - '_B' + str(self.boxtype.upper())[0] + \ - '_Z127' + \ - '_P' + str(self.padding) + \ - '_LN' + str(self.lmin) + \ - '_LX' + str(self.lmax) + \ - '_O' + str(self.overlap) + \ - '_NV' + str(self.nrvir) - - self.halopath = self.gadpath + 'halos/H' + str(self.haloid) + '/' + self.foldername + "/rockstardata" + self.foldername = naming.run_dir_name(halo_id=self.haloid, **self._run_kwargs()) + self.halopath = naming.rockstar_dir(self.gadpath, self.haloid, **self._run_kwargs()) if os.path.exists(self.halopath): halodata = RSDataReader.RSDataReader(self.halopath,self.snapshot,digits=2) @@ -240,16 +243,8 @@ def _plot_button_fired(self): dataexists = False elif self.parentorzoom == 'zoom': - self.foldername = 'H' + str(self.haloid) + \ - '_B' + str(self.boxtype.upper())[0] + \ - '_Z127' + \ - '_P' + str(self.padding) + \ - '_LN' + str(self.lmin) + \ - '_LX' + str(self.lmax) + \ - '_O' + str(self.overlap) + \ - '_NV' + str(self.nrvir) - - self.halopath = self.gadpath + 'halos/H' + str(self.haloid) + '/' + self.foldername + "/rockstardata" + self.foldername = naming.run_dir_name(halo_id=self.haloid, **self._run_kwargs()) + self.halopath = naming.rockstar_dir(self.gadpath, self.haloid, **self._run_kwargs()) if os.path.exists(self.halopath): dataexists = True diff --git a/src/cme/ui/ics.py b/src/cme/ui/ics.py index 10f9ca5..493799e 100644 --- a/src/cme/ui/ics.py +++ b/src/cme/ui/ics.py @@ -1,4 +1,6 @@ from cme.ui.common import * + +from cme import naming from glob import glob import random @@ -265,17 +267,12 @@ def _generate_button_fired(self): for lmini in self.lmin: for lmaxi in self.lmax: for overlapi in self.overlap: - foldername = 'H' + str(self.haloidselect) + \ - '_B' + str(boxtypei.upper())[0] + \ - '_Z' + str(self.zinit) + \ - '_P' + str(paddingi) + \ - '_LN' + str(lmini) + \ - '_LX' + str(lmaxi) + \ - '_O' + str(overlapi) + \ - '_NV' + str(nrviri) - - #dirtest = "/n/home01/bgriffen/data/caterpillar/" - filepath = self.outpath + 'halos/H' + str(self.haloidselect) + '/' + foldername + run = dict(region=boxtypei, padding=paddingi, + levelmin=lmini, levelmax=lmaxi, + overlap=overlapi, nrvir=nrviri, + zinit=self.zinit) + foldername = naming.run_dir_name(halo_id=self.haloidselect, **run) + filepath = naming.run_dir(self.outpath, self.haloidselect, **run) if os.path.exists(filepath): self.confstatus = 'DIR EXIST!' @@ -286,14 +283,16 @@ def _generate_button_fired(self): #pointfile = dirtest + 'ics/lagr/H' + str(self.haloidselect) + 'NRVIR' + str(int(nrviri)) #pointfile2 = dirtest + 'ics/lagr/H' + str(self.haloidselect) + 'NRVIR' + str(int(nrviri)) - pointfile = self.outpath + 'ics/lagr/H' + str(self.haloidselect) + 'NRVIR' + str(int(nrviri)) + pointfile = naming.lagr_region_path( + os.path.join(self.outpath, 'ics', 'lagr'), + self.haloidselect, nrviri) #if lmaxi < 13: # pointfile2 = pointfile writepath = filepath #self.outpath + 'halos/' + foldername - confname = filepath + '/' + foldername + '.conf' + confname = os.path.join(filepath, foldername + '.conf') #self.outpath + 'halos/' + foldername + '/' + foldername + '.conf' self.centx,self.centy,self.centz,self.extx,self.exty,self.extz = getcentext(pointfile + '.head') @@ -370,7 +369,7 @@ def _resimlagrfile_changed(self): def _existencebutton_fired(self): if len(self.nrvir) == 1: - filename = self.lagroutputdir + '/H' + str(self.haloidselect) + 'NRVIR' + str(int(self.nrvir[0])) + filename = naming.lagr_region_path(self.lagroutputdir, self.haloidselect, self.nrvir[0]) try: with open(filename): self.filestatus = 'Lagrangian file exists.' @@ -446,7 +445,7 @@ def _projopt_changed(self): def _makeic_button_fired(self): for Nrvir in self.nrvir: - filename = self.lagroutputdir + '/H' + str(self.haloidselect) + 'NRVIR' + str(int(Nrvir)) + filename = naming.lagr_region_path(self.lagroutputdir, self.haloidselect, Nrvir) try: with open(filename): if len(self.nrvir) == 1: @@ -662,7 +661,8 @@ def _makeic_button_fired(self): self.zpos = lagrPos[:,2] if self.writelagrfile == True: - headerfilename = self.lagroutputdir + '/H' + str(self.haloidselect) + 'NRVIR' + str(int(Nrvir)) + '.head' + headerfilename = naming.lagr_header_path( + self.lagroutputdir, self.haloidselect, Nrvir) f1=open(headerfilename,'w') f1.write('#' + str(self.centx) + '\n') f1.write('#' + str(self.centy) + '\n') @@ -672,7 +672,8 @@ def _makeic_button_fired(self): f1.write('#' + str(self.extz) + '\n') f1.close() - filename = self.lagroutputdir + '/H' + str(self.haloidselect) + 'NRVIR' + str(int(Nrvir)) + filename = naming.lagr_region_path( + self.lagroutputdir, self.haloidselect, Nrvir) f2=open(filename,'w') for iv in range(0,len(lagrPos[:,0])): f2.write(str(lagrPos[iv,0]/header.boxsize)+' '+str(lagrPos[iv,1]/header.boxsize)+' '+ str(lagrPos[iv,2]/header.boxsize)+'\n') diff --git a/tests/test_naming.py b/tests/test_naming.py new file mode 100644 index 0000000..b751f37 --- /dev/null +++ b/tests/test_naming.py @@ -0,0 +1,149 @@ +import pytest + +from cme import naming + + +class TestRegionCode: + def test_known_regions(self): + assert naming.region_code("box") == "B" + assert naming.region_code("ellipsoid") == "E" + + def test_is_case_insensitive(self): + # The GUI passes Enum values through .upper() in some tabs, not others. + assert naming.region_code("Ellipsoid") == "E" + assert naming.region_code("BOX") == "B" + + def test_rejects_unknown_region(self): + with pytest.raises(ValueError, match="unknown region"): + naming.region_code("sphere") + + +class TestRunDirName: + def test_matches_the_established_convention(self): + # A real Caterpillar run directory, as it appears in the screenshots. + assert ( + naming.run_dir_name( + halo_id=190897, + region="ellipsoid", + padding=7, + levelmin=7, + levelmax=11, + overlap=4, + nrvir=4, + ) + == "H190897_BE_Z127_P7_LN7_LX11_O4_NV4" + ) + + def test_box_region_uses_b(self): + name = naming.run_dir_name( + halo_id=1930, region="box", padding=8, levelmin=7, + levelmax=12, overlap=4, nrvir=3, + ) + assert name == "H1930_BB_Z127_P8_LN7_LX12_O4_NV3" + + def test_accepts_strings_as_well_as_ints(self): + # The ICs and Gadget tabs supply these as strings from a + # CheckListEditor; the contamination and halo tabs supply ints. + from_strings = naming.run_dir_name( + halo_id="190897", region="ellipsoid", padding="7", levelmin="7", + levelmax="11", overlap="4", nrvir="4", + ) + from_ints = naming.run_dir_name( + halo_id=190897, region="ellipsoid", padding=7, levelmin=7, + levelmax=11, overlap=4, nrvir=4, + ) + assert from_strings == from_ints + + def test_zinit_defaults_to_127_but_is_honoured(self): + # The contamination and halo tabs previously hardcoded _Z127, so a + # non-default zinit silently pointed them at a directory that the ICs + # tab had never created. + assert "_Z127_" in naming.run_dir_name( + halo_id=1, region="box", padding=7, levelmin=7, + levelmax=11, overlap=4, nrvir=1, + ) + assert "_Z63_" in naming.run_dir_name( + halo_id=1, region="box", padding=7, levelmin=7, + levelmax=11, overlap=4, nrvir=1, zinit=63, + ) + + +class TestParseRunDirName: + def test_round_trips(self): + kwargs = dict( + halo_id=241932, region="box", padding=9, levelmin=8, + levelmax=13, overlap=5, nrvir=7, zinit=63, + ) + parsed = naming.parse_run_dir_name(naming.run_dir_name(**kwargs)) + assert parsed == kwargs + + def test_returns_none_for_unrelated_names(self): + assert naming.parse_run_dir_name("outputs") is None + assert naming.parse_run_dir_name("H190897_BE_Z127") is None + + +class TestSnapshotIndex: + @pytest.mark.parametrize( + "snapnum,expected", + [(0, "000"), (5, "005"), (63, "063"), (64, "064"), (127, "127"), (255, "255")], + ) + def test_pads_to_three_digits(self, snapnum, expected): + assert naming.snapshot_index(snapnum) == expected + + def test_regression_single_digit_snapshots(self): + # The inline form was "groups_0" + str(snapnum), which produced + # "groups_05" for snapshot 5. Gadget writes "groups_005", so every + # existence check against a single-digit snapshot reported "not found". + assert naming.groups_dir("/out", 5).endswith("groups_005") + assert naming.snapshot_dir("/out", 5).endswith("snapdir_005") + + def test_regression_three_digit_snapshots(self): + # Likewise the zoom tabs default to snapshot 255, where the inline + # form produced "snapdir_0255". + assert naming.snapshot_dir("/out", 255).endswith("snapdir_255") + + +class TestPaths: + RUN = dict( + region="ellipsoid", padding=7, levelmin=7, + levelmax=11, overlap=4, nrvir=4, + ) + + def test_run_dir(self): + assert naming.run_dir("/data", 190897, **self.RUN) == ( + "/data/halos/H190897/H190897_BE_Z127_P7_LN7_LX11_O4_NV4" + ) + + def test_outputs_and_children(self): + outputs = naming.outputs_dir("/data", 190897, **self.RUN) + assert outputs.endswith("/H190897_BE_Z127_P7_LN7_LX11_O4_NV4/outputs") + assert naming.groups_dir(outputs, 63).endswith("/outputs/groups_063") + assert naming.snapshot_base(outputs, 63).endswith( + "/outputs/snapdir_063/snap_063" + ) + + def test_rockstar_dir(self): + assert naming.rockstar_dir("/data", 190897, **self.RUN).endswith( + "/H190897_BE_Z127_P7_LN7_LX11_O4_NV4/rockstardata" + ) + + def test_trailing_slash_on_base_is_normalised(self): + # The inline form was `self.gadpath + 'halos/H'`, which silently + # produced ".../datahalos/H190897" when the path trait had no + # trailing slash. + assert naming.run_dir("/data/", 190897, **self.RUN) == naming.run_dir( + "/data", 190897, **self.RUN + ) + + def test_lagrangian_paths(self): + assert naming.lagr_region_path("/d/ics/lagr", 190897, 4) == ( + "/d/ics/lagr/H190897NRVIR4" + ) + assert naming.lagr_header_path("/d/ics/lagr", 190897, 4) == ( + "/d/ics/lagr/H190897NRVIR4.head" + ) + + def test_lagrangian_paths_coerce_nrvir(self): + # The ICs tab passes nrvir as a one-element list of strings; formatting + # it without coercion produced "H190897NRVIR['4']". + assert naming.lagr_region_path("/d", 190897, "4") == "/d/H190897NRVIR4" From 3911c05af3f735d4959ba2f0682d8b366b3d2d83 Mon Sep 17 00:00:00 2001 From: Brendan Griffen Date: Mon, 27 Jul 2026 22:04:34 +1000 Subject: [PATCH 08/19] refactor: extract named cosmologies into cme.cosmology The Hubble parameter appeared as six hardcoded 0.6711 literals across four tabs, and the full parameter sets came from `grifflib.cosmoconstant` in the external `modules` package that is not distributed with this repository. A value duplicated six times is a correctness risk in a codebase whose output is scientific, and depending on an absent module for it made the values unauditable. cme/cosmology.py holds one frozen dataclass per set (WMAP1/3/5/7/9, PLANCK) with a literature citation on each, and a `cosmoconstant()` shim preserving the old six-tuple signature so call sites are unchanged in shape. On the values: `grifflib` is absent, so these were taken from the cited papers rather than copied from the original. The PLANCK row cross-checks exactly against the values hardcoded as defaults in the Gadget tab (Omega0 = 0.3175, OmegaLambda = 0.6825, h = 0.6711), and its Omega_b agrees with Planck 2013's Omega_b h^2 = 0.02205 to three decimal places, which is good evidence the conventions match. This is called out in the module docstring: if the original grifflib is recovered, the non-PLANCK rows should be diffed against it before being trusted for new science. Both `h` and `H0` are exposed because the two output formats disagree -- MUSIC configs want H0 in km/s/Mpc, Gadget parameter files want h -- and the old tuple carried H0 while the Gadget tab's own default carried h. Tests pin that convention so the ambiguity cannot silently flip; the resulting double-conversion in the Gadget writer is addressed in the next commit. Tests assert flatness, physical plausibility and a citation for every set. --- src/cme/cosmology.py | 126 ++++++++++++++++++++++++++++++++++++ src/cme/ui/candidates.py | 6 +- src/cme/ui/common.py | 2 - src/cme/ui/contamination.py | 8 +-- src/cme/ui/gadget.py | 4 +- src/cme/ui/halos.py | 4 +- src/cme/ui/ics.py | 6 +- tests/test_cosmology.py | 81 +++++++++++++++++++++++ 8 files changed, 222 insertions(+), 15 deletions(-) create mode 100644 src/cme/cosmology.py create mode 100644 tests/test_cosmology.py diff --git a/src/cme/cosmology.py b/src/cme/cosmology.py new file mode 100644 index 0000000..eed178c --- /dev/null +++ b/src/cme/cosmology.py @@ -0,0 +1,126 @@ +"""Named cosmological parameter sets. + +This replaces ``cosmoconstant()`` from the external ``grifflib`` module, which +is not distributed with this repository, and the six hardcoded copies of the +Hubble parameter that were scattered through the GUI tabs. + +.. warning:: + + Because ``grifflib`` is absent, the values below were taken from the + published papers cited against each set rather than copied from the + original implementation. The PLANCK row cross-checks exactly against the + values hardcoded in the Gadget tab (Omega0 = 0.3175, OmegaLambda = 0.6825, + h = 0.6711), which is good evidence the convention matches. If the original + ``grifflib`` is recovered, diff it against this table before trusting a + non-PLANCK set for new science. + +``h`` is the dimensionless Hubble parameter; ``H0`` is in km/s/Mpc. MUSIC +configuration files want ``H0``, Gadget parameter files want ``h``, which is +why both are exposed rather than one being derived at each call site. +""" + +from dataclasses import dataclass + + +@dataclass(frozen=True) +class Cosmology: + """One set of cosmological parameters.""" + + name: str + omega_m: float + omega_l: float + omega_b: float + h: float + sigma_8: float + n_spec: float + reference: str + + @property + def H0(self): + """Hubble constant in km/s/Mpc, as MUSIC's ``H0`` key expects.""" + return self.h * 100.0 + + def as_tuple(self): + """The 6-tuple the GUI tabs unpack, in ``cosmoconstant()``'s order. + + Note the fourth element is ``H0`` in km/s/Mpc, not ``h``: the Gadget + tab divides it by 100 when writing ``HubbleParam``, and the MUSIC + writer uses it directly. + """ + return ( + self.omega_m, + self.omega_l, + self.omega_b, + self.H0, + self.sigma_8, + self.n_spec, + ) + + +WMAP1 = Cosmology( + name="WMAP1", omega_m=0.270, omega_l=0.730, omega_b=0.0440, + h=0.710, sigma_8=0.840, n_spec=0.990, + reference="Spergel et al. 2003, ApJS 148, 175", +) + +WMAP3 = Cosmology( + name="WMAP3", omega_m=0.234, omega_l=0.766, omega_b=0.0407, + h=0.734, sigma_8=0.760, n_spec=0.961, + reference="Spergel et al. 2007, ApJS 170, 377", +) + +WMAP5 = Cosmology( + name="WMAP5", omega_m=0.258, omega_l=0.742, omega_b=0.0441, + h=0.719, sigma_8=0.796, n_spec=0.963, + reference="Komatsu et al. 2009, ApJS 180, 330", +) + +WMAP7 = Cosmology( + name="WMAP7", omega_m=0.272, omega_l=0.728, omega_b=0.0455, + h=0.704, sigma_8=0.807, n_spec=0.961, + reference="Komatsu et al. 2011, ApJS 192, 18", +) + +WMAP9 = Cosmology( + name="WMAP9", omega_m=0.2865, omega_l=0.7135, omega_b=0.0463, + h=0.6932, sigma_8=0.820, n_spec=0.9608, + reference="Hinshaw et al. 2013, ApJS 208, 19", +) + +PLANCK = Cosmology( + name="PLANCK", omega_m=0.3175, omega_l=0.6825, omega_b=0.0490, + h=0.6711, sigma_8=0.8344, n_spec=0.9624, + reference="Planck Collaboration 2014, A&A 571, A16 (Planck 2013 XVI)", +) + +#: Keyed by the names the GUI's CheckListEditor offers. +COSMOLOGIES = { + c.name: c for c in (WMAP1, WMAP3, WMAP5, WMAP7, WMAP9, PLANCK) +} + +#: The Caterpillar suite was run with Planck 2013. Tabs that analyse existing +#: output and have no cosmology selector of their own assume this, which is +#: what the six hardcoded 0.6711 literals were doing implicitly. +DEFAULT = PLANCK + + +def get(name): + """Look up a cosmology by name, case-insensitively. + + >>> get("planck").h + 0.6711 + """ + try: + return COSMOLOGIES[str(name).upper()] + except KeyError: + raise ValueError( + f"unknown cosmology {name!r}; expected one of {sorted(COSMOLOGIES)}" + ) from None + + +def cosmoconstant(name): + """Compatibility shim matching the old ``grifflib.cosmoconstant`` signature. + + Returns ``(omega_m, omega_l, omega_b, H0, sigma_8, n_spec)``. + """ + return get(name).as_tuple() diff --git a/src/cme/ui/candidates.py b/src/cme/ui/candidates.py index 458d153..d22cf58 100644 --- a/src/cme/ui/candidates.py +++ b/src/cme/ui/candidates.py @@ -1,4 +1,6 @@ from cme.ui.common import * + +from cme import cosmology from numpy.random import random class Candidates(HasTraits): @@ -148,7 +150,7 @@ def _clear_button_fired(self): self.main.mergertreetab.initstab = [] def _getcandidates_button_fired(self): - hubble = 0.6711 + hubble = cosmology.DEFAULT.h if self.loadpreviouscand == False: halodata = RSDataReader.RSDataReader(self.halopath,self.snapnum,digits=2) allhalos = halodata.get_hosts() @@ -302,7 +304,7 @@ def __init__(self, main, **kwargs): HasTraits.__init__(self) self.main = main self.exclusionstr = 'Exclusion Zones: "no halo with mass M within distance R"' - self.hubble = 0.6711 + self.hubble = cosmology.DEFAULT.h #self.haloid = [190897] self.lowermasscut = np.log10(7e11) diff --git a/src/cme/ui/common.py b/src/cme/ui/common.py index b82086e..801cd28 100644 --- a/src/cme/ui/common.py +++ b/src/cme/ui/common.py @@ -55,7 +55,6 @@ COM, CorrectPos, addsubtitle, - cosmoconstant, drawcircle, getcandidatelist, getcentext, @@ -112,7 +111,6 @@ "COM", "CorrectPos", "addsubtitle", - "cosmoconstant", "drawcircle", "getcandidatelist", "getcentext", diff --git a/src/cme/ui/contamination.py b/src/cme/ui/contamination.py index ce9b7ee..11e881a 100644 --- a/src/cme/ui/contamination.py +++ b/src/cme/ui/contamination.py @@ -1,7 +1,7 @@ from cme.ui.common import * import modules.mergertrees.MTCatalogue as MT -from cme import naming +from cme import cosmology, naming class Contamination(HasTraits): @@ -84,7 +84,7 @@ class Contamination(HasTraits): def _checkcontam_button_fired(self): self.foldername = naming.run_dir_name(halo_id=self.haloid, **self._run_kwargs()) - hubble = 0.6711 + hubble = cosmology.DEFAULT.h if os.path.exists(self.groups_dir()): tmppath = self.outputs_dir() @@ -272,7 +272,7 @@ def _plothalodist_button_fired(self): ax = figure.add_subplot(111) ax = self.main.display.axes[0] - hubble = 0.6711 + hubble = cosmology.DEFAULT.h if os.path.exists(self.groups_dir()): self.datastatus = "Data exists, plotted." @@ -339,7 +339,7 @@ def _plothalodist_button_fired(self): def _plotmayavi_button_fired(self): self.main.scene.mlab.clf(figure=self.main.scene.mayavi_scene) self.foldername = naming.run_dir_name(halo_id=self.haloid, **self._run_kwargs()) - hubble = 0.6711 + hubble = cosmology.DEFAULT.h if os.path.exists(self.groups_dir()): self.datastatus = "Data exists, plotted." tmppath = self.outputs_dir() diff --git a/src/cme/ui/gadget.py b/src/cme/ui/gadget.py index 220d38c..3c92412 100644 --- a/src/cme/ui/gadget.py +++ b/src/cme/ui/gadget.py @@ -1,6 +1,6 @@ from cme.ui.common import * -from cme import naming +from cme import cosmology, naming import modules.mergertrees.MTCatalogue as MT class GadgetRun(HasTraits): @@ -287,7 +287,7 @@ def _subscript_button_fired(self): else: print("PATH FOUND:",filepath) self.levelmaxuse = int(self.lmax) - self.Omega0,self.OmegaLambda,self.OmegaBaryon,self.HubbleParam,sigma8,nspec = cosmoconstant(self.cosmologylist[0]) + self.Omega0,self.OmegaLambda,self.OmegaBaryon,self.HubbleParam,sigma8,nspec = cosmology.cosmoconstant(self.cosmologylist[0]) ext = self.gadpath + 'halos/H' + str(self.haloid) + '/' + foldername if self.ENABLE_SUBFIND == True: self.makeparam('param.txt',ext,includesub=False) diff --git a/src/cme/ui/halos.py b/src/cme/ui/halos.py index 489e8ce..c0f49f9 100644 --- a/src/cme/ui/halos.py +++ b/src/cme/ui/halos.py @@ -1,6 +1,6 @@ from cme.ui.common import * -from cme import naming +from cme import cosmology, naming class HaloFind(HasTraits): @@ -392,7 +392,7 @@ def _plotxyzrvir_button_changed(self): rvmax = np.array(halodata.data['rvmax'][condnew]) vmax = np.array(halodata.data['vmax'][condnew]) - print(mvir,mvir/0.6711,rvirn,rvmax,vmax,xposhost,yposhost,zposhost) + print(mvir,mvir/cosmology.DEFAULT.h,rvirn,rvmax,vmax,xposhost,yposhost,zposhost) extent = [xposhost-boxwidth,xposhost+boxwidth,yposhost-boxwidth,yposhost+boxwidth,zposhost-boxwidth,zposhost+boxwidth] self.main.scene.mlab.points3d(x, y, z, rvir/1000,colormap="copper") diff --git a/src/cme/ui/ics.py b/src/cme/ui/ics.py index 493799e..87774ac 100644 --- a/src/cme/ui/ics.py +++ b/src/cme/ui/ics.py @@ -1,6 +1,6 @@ from cme.ui.common import * -from cme import naming +from cme import cosmology, naming from glob import glob import random @@ -235,7 +235,7 @@ def _generate_button_fired(self): confname = self.outpath + 'parent/' + foldername + '/' + foldername + '.conf' - omegam,omegal,omegab,hubble,sigma8,nspec = cosmoconstant(cosmi) + omegam,omegal,omegab,hubble,sigma8,nspec = cosmology.cosmoconstant(cosmi) baryonsstr = determineboolstr(self.baryons) use2LPTstr = determineboolstr(self.use2LPT) @@ -299,7 +299,7 @@ def _generate_button_fired(self): #print self.parentbox #print self.centx,self.centy,self.centz #print self.extx,self.exty,self.extz - omegam,omegal,omegab,hubble,sigma8,nspec = cosmoconstant(cosmi) + omegam,omegal,omegab,hubble,sigma8,nspec = cosmology.cosmoconstant(cosmi) baryonsstr = determineboolstr(self.baryons) use2LPTstr = determineboolstr(self.use2LPT) diff --git a/tests/test_cosmology.py b/tests/test_cosmology.py new file mode 100644 index 0000000..490231b --- /dev/null +++ b/tests/test_cosmology.py @@ -0,0 +1,81 @@ +import pytest + +from cme import cosmology + + +class TestLookup: + def test_get_is_case_insensitive(self): + assert cosmology.get("planck") is cosmology.PLANCK + assert cosmology.get("PLANCK") is cosmology.PLANCK + assert cosmology.get("WMAP7") is cosmology.WMAP7 + + def test_rejects_unknown_name(self): + with pytest.raises(ValueError, match="unknown cosmology"): + cosmology.get("WMAP11") + + def test_every_name_offered_by_the_gui_resolves(self): + # These are the values in the CheckListEditor on the ICs and Gadget tabs. + for name in ("WMAP1", "WMAP3", "WMAP5", "WMAP7", "WMAP9", "PLANCK"): + assert cosmology.get(name).name == name + + +class TestPlanckCrossCheck: + """PLANCK is the row that can be verified against the existing source. + + The Gadget tab hardcoded these same values as its defaults, so they are a + reliable check that this table follows the same conventions as the + `grifflib.cosmoconstant` it replaces. + """ + + def test_matches_the_hardcoded_gadget_defaults(self): + planck = cosmology.PLANCK + assert planck.omega_m == 0.3175 + assert planck.omega_l == 0.6825 + assert planck.h == 0.6711 + + def test_omega_b_is_consistent_with_planck_2013_omega_b_h_squared(self): + # Planck 2013 XVI reports Omega_b h^2 = 0.02205. + assert cosmology.PLANCK.omega_b == pytest.approx( + 0.02205 / cosmology.PLANCK.h**2, abs=1e-3 + ) + + +class TestInternalConsistency: + @pytest.mark.parametrize("name", sorted(cosmology.COSMOLOGIES)) + def test_flat_universe(self, name): + c = cosmology.get(name) + assert c.omega_m + c.omega_l == pytest.approx(1.0, abs=1e-3) + + @pytest.mark.parametrize("name", sorted(cosmology.COSMOLOGIES)) + def test_parameters_are_physically_plausible(self, name): + c = cosmology.get(name) + assert 0.0 < c.omega_b < c.omega_m + assert 0.5 < c.h < 1.0 + assert 0.5 < c.sigma_8 < 1.2 + assert 0.9 < c.n_spec < 1.1 + + @pytest.mark.parametrize("name", sorted(cosmology.COSMOLOGIES)) + def test_every_set_cites_a_reference(self, name): + assert cosmology.get(name).reference + + +class TestUnitConvention: + def test_H0_is_in_km_per_s_per_mpc(self): + assert cosmology.PLANCK.H0 == pytest.approx(67.11) + + def test_as_tuple_returns_H0_not_h(self): + # The Gadget tab writes `HubbleParam` as this value / 100, so the tuple + # must carry H0. Returning h here would emit HubbleParam = 0.006711. + _, _, _, hubble, _, _ = cosmology.PLANCK.as_tuple() + assert hubble == pytest.approx(67.11) + assert hubble / 100 == pytest.approx(cosmology.PLANCK.h) + + def test_cosmoconstant_shim_matches_the_old_six_tuple_order(self): + omega_m, omega_l, omega_b, hubble, sigma_8, n_spec = ( + cosmology.cosmoconstant("PLANCK") + ) + assert (omega_m, omega_l, omega_b) == (0.3175, 0.6825, 0.0490) + assert (hubble, sigma_8, n_spec) == pytest.approx((67.11, 0.8344, 0.9624)) + + def test_default_is_planck(self): + assert cosmology.DEFAULT is cosmology.PLANCK From 784a13ee6b7fb9c20a7b9421abc344e8a660175d Mon Sep 17 00:00:00 2001 From: Brendan Griffen Date: Mon, 27 Jul 2026 22:15:43 +1000 Subject: [PATCH 09/19] refactor: extract the configuration writers into cme.writers MUSIC configs, Gadget parameter files, Config.sh and the batch submission scripts were all built by long runs of sequential f.write calls interleaved with GUI state -- roughly 110 writes for param.txt, 90 for Config.sh, 50 per MUSIC conf. None of it could be tested, so a malformed parameter file would only surface once it had consumed a cluster allocation. cme/writers/ holds pure functions that take scalars and return text: music.py resim_conf, parent_conf, refinement_seeds gadget.py param_file, config_sh, expansion_list, softenings, pmgrid_for_levelmax submit.py slurm_script, pbs_script, mpirun_command Tests parse the output back with configparser (MUSIC) and a key/value splitter (Gadget) rather than only string-matching, plus a byte-exact golden file for a MUSIC resim conf. Fixes carried by the extraction: - AUDIT 2.1.1: SLURM submission raised TypeError, concatenating integer traits with strings ('#SBATCH -n ' + self.SLURMcores). SLURM had never worked. - AUDIT 2.2.5: the SLURM mpirun line was built from the *PBS* core count, so a SLURM job would have launched with the wrong number of ranks. - AUDIT 2.2.6: both branches of _SLURMcores_changed were identical, so enabling SUBFIND left SLURM pointing at the plain binary and param.txt while PBS correctly switched to P-Gadget3_sub and param_sub.txt. - AUDIT 2.1.11: determineboolstr returned an unbound local for anything not exactly True or False (a numpy bool, say). Replaced by a total function. - AUDIT 2.1.12: the run script was opened before its directory was known to exist; writers.write now creates parents and is called once the path is known. - HubbleParam was stored as H0 in a trait named for h and divided by 100 when written, so the value displayed in the GUI disagreed with the value on disk. The writer takes the cosmology and emits h directly. - Config.sh emitted `TOKEN# comment` with no separating space for tokens longer than the comment column, e.g. FOF_SECONDARY_LINK_TYPES=4+8+16+32. Found by a test asserting the separation, now guaranteed. Deduplication: the 6 handlers that rebuilt the mpirun line held 18 copies of it between them and are now one execute_command(); the 12-line softening cascade appeared twice and the PMGRID thresholds once, all now in the writer. ics.py loses 124 lines, gadget.py 215. Whitespace differs from the old output in one respect: seed[10] and above were previously misaligned by one column because the padding was a fixed string. MUSIC's parser is whitespace-insensitive, so this is cosmetic. --- src/cme/ui/gadget.py | 471 ++++++++++------------------------- src/cme/ui/ics.py | 266 ++++++-------------- src/cme/writers/__init__.py | 22 ++ src/cme/writers/gadget.py | 443 ++++++++++++++++++++++++++++++++ src/cme/writers/music.py | 258 +++++++++++++++++++ src/cme/writers/submit.py | 142 +++++++++++ tests/test_writers_gadget.py | 204 +++++++++++++++ tests/test_writers_music.py | 205 +++++++++++++++ tests/test_writers_submit.py | 153 ++++++++++++ 9 files changed, 1633 insertions(+), 531 deletions(-) create mode 100644 src/cme/writers/__init__.py create mode 100644 src/cme/writers/gadget.py create mode 100644 src/cme/writers/music.py create mode 100644 src/cme/writers/submit.py create mode 100644 tests/test_writers_gadget.py create mode 100644 tests/test_writers_music.py create mode 100644 tests/test_writers_submit.py diff --git a/src/cme/ui/gadget.py b/src/cme/ui/gadget.py index 3c92412..d5f2295 100644 --- a/src/cme/ui/gadget.py +++ b/src/cme/ui/gadget.py @@ -1,6 +1,7 @@ from cme.ui.common import * -from cme import cosmology, naming +from cme import cosmology, naming, writers +from cme.writers import gadget as gadgetwriter, submit import modules.mergertrees.MTCatalogue as MT class GadgetRun(HasTraits): @@ -303,60 +304,43 @@ def _subscript_button_fired(self): mkdirs = "mkdir -p " + filepath + "/outputs" subprocess.call([mkdirs], shell=True) - if self.ENABLE_SUBFIND == True: - f = open(filepath + "/runscript_sub",'w') - else: - f = open(filepath + "/runscript",'w') - #f = open(filepath + "/runscript",'w') + scriptpath = os.path.join( + filepath, + "runscript_sub" if self.ENABLE_SUBFIND else "runscript") if self.subtype == "SLURM": - f.write('#!/bin/bash\n') - f.write('#SBATCH -n ' + self.SLURMcores + '\n') - f.write('#SBATCH -o ' + jobname + '.o%j\n') - f.write('#SBATCH -e ' + jobname + '.e%j\n') - f.write('#SBATCH -t ' + self.SLURMtime + '\n') - f.write('#SBATCH -p ' + self.SLURMqueue + '\n') - f.write('#SBATCH --mem-per-cpu=' + self.SLURMmemory + '\n') - f.write('#SBATCH --mail-user=' + self.emailaddress + '\n') - f.write('#SBATCH -J '+ jobname + '\n') - f.write('#SBATCH --mail-type=begin\n') - f.write('#SBATCH --mail-type=end\n') - f.write("\n") - f.write("cd " + filepath + "\n") - f.write("\n") - self.SLURMexecute = 'mpirun -np ' + str(self.PBSncores) + './P-Gadget3 param.txt 1>OUTPUT 2>ERROR\n' - f.write(self.SLURMexecute +'\n') - f.close() - - if self.subtype == "PBS": - f.write("#!/bin/csh" + "\n") - f.write("#PBS -k eo\n") - f.write("#PBS -l nodes=" + str(self.PBSnnodes) + ":ppn=" + str(self.PBSncores) + "\n") - f.write("#PBS -N " + jobname + "\n") - f.write("#PBS -M " + self.emailaddress + "\n") - f.write("#PBS -o " + filepath + "/" + jobname + ".out\n") - f.write("#PBS -e " + filepath + "/" + jobname + ".err\n") - #f.write("#PBS -u " + self.username + "\n") - f.write("#PBS -q " + self.PBSqueue + "\n") - f.write(self.PBSextraflags + "\n") - f.write("\n") - f.write(self.PBSextralines + "\n") - f.write("\n") - f.write("cd " + filepath + "\n") - f.write("\n") - f.write(self.PBSexecute) - f.close() + script = submit.slurm_script( + job_name=jobname, + work_dir=filepath, + execute=self.execute_command(), + cores=self.SLURMcores, + minutes=self.SLURMtime, + memory_mb=self.SLURMmemory, + queue=self.SLURMqueue, + email=self.emailaddress) + else: + script = submit.pbs_script( + job_name=jobname, + work_dir=filepath, + execute=self.execute_command(), + cores_per_node=self.PBSncores, + nodes=self.PBSnnodes, + queue=self.PBSqueue, + email=self.emailaddress, + extra_flags=self.PBSextraflags, + extra_lines=self.PBSextralines) + writers.write(scriptpath, script) if self.compilegadopt == True: if self.ENABLE_SUBFIND == True: - self.constructconfigsh(filepath + "/Config_sub.sh") + self.write_config_sh(os.path.join(filepath, "Config_sub.sh")) command5 = "tail -n+96 " + gadgetfilepath + "P-Gadget3/Config.sh > " + filepath + "/bottomConfig" command6 = "cat " + filepath + "/bottomConfig >> " + filepath + "/Config_sub.sh" command7 = "cp " + filepath + "/Config_sub.sh " + gadgetfilepath + "P-Gadget3/Config.sh" subprocess.call(';'.join([command5,command6,command7]), shell=True) else: - self.constructconfigsh(filepath + "/Config.sh") + self.write_config_sh(os.path.join(filepath, "Config.sh")) command5 = "tail -n+96 " + gadgetfilepath + "/P-Gadget3/Config.sh > " + filepath + "/bottomConfig" command6 = "cat " + filepath + "/bottomConfig >> " + filepath + "/Config.sh" command7 = "cp " + filepath + "/Config.sh " + gadgetfilepath + "P-Gadget3/Config.sh" @@ -405,49 +389,35 @@ def _subscript_button_fired(self): print("SUBMITTED JOBS!") - def _ENABLE_SUBFIND_changed(self): - if self.ENABLE_SUBFIND == True: - self.PBSexecute = "mpirun -np " + str(int(self.PBSncores*self.PBSnnodes)) + " ./P-Gadget3_sub ./param_sub.txt 3 63 1>" + self.outfile + "sub 2>" + self.errorfile + "sub" - self.SLURMexecute = "mpirun -np " + str(int(self.SLURMcores)) + " ./P-Gadget3_sub ./param_sub.txt 3 63 1>" + self.outfile + "sub 2>" + self.errorfile + "sub" - else: - self.PBSexecute = "mpirun -np " + str(int(self.PBSncores*self.PBSnnodes)) + " ./P-Gadget3 ./param.txt 1>" + self.outfile + " 2>" + self.errorfile - self.SLURMexecute = "mpirun -np " + str(int(self.SLURMcores)) + " ./P-Gadget3 ./param.txt 1>" + self.outfile + " 2>" + self.errorfile - - def _PBSncores_changed(self): - if self.ENABLE_SUBFIND == True: - self.PBSexecute = "mpirun -np " + str(int(self.PBSncores*self.PBSnnodes)) + " ./P-Gadget3_sub ./param_sub.txt 3 63 1>" + self.outfile + "sub 2>" + self.errorfile + "sub" - else: - self.PBSexecute = "mpirun -np " + str(int(self.PBSncores*self.PBSnnodes)) + " ./P-Gadget3 ./param.txt 1>" + self.outfile + " 2>" + self.errorfile - - def _PBSnnodes_changed(self): - if self.ENABLE_SUBFIND == True: - self.PBSexecute = "mpirun -np " + str(int(self.PBSncores*self.PBSnnodes)) + " ./P-Gadget3_sub ./param_sub.txt 3 63 1>" + self.outfile + "sub 2>" + self.errorfile + "sub" - else: - self.PBSexecute = "mpirun -np " + str(int(self.PBSncores*self.PBSnnodes)) + " ./P-Gadget3 ./param.txt 1>" + self.outfile + " 2>" + self.errorfile - - def _SLURMcores_changed(self): - if self.ENABLE_SUBFIND == True: - self.SLURMexecute = "mpirun -np " + str(int(self.SLURMcores)) + " ./P-Gadget3 ./param.txt 1>" + self.outfile + " 2>" + self.errorfile - else: - self.SLURMexecute = "mpirun -np " + str(int(self.SLURMcores)) + " ./P-Gadget3 ./param.txt 1>" + self.outfile + " 2>" + self.errorfile - - def _errorfile_changed(self): - if self.ENABLE_SUBFIND == True: - self.PBSexecute = "mpirun -np " + str(int(self.PBSncores*self.PBSnnodes)) + " ./P-Gadget3_sub ./param_sub.txt 3 63 1>" + self.outfile + "sub 2>" + self.errorfile + "sub" - self.SLURMexecute = "mpirun -np " + str(int(self.SLURMcores)) + " ./P-Gadget3_sub ./param_sub.txt 3 63 1>" + self.outfile + "sub 2>" + self.errorfile + "sub" - else: - self.PBSexecute = "mpirun -np " + str(int(self.PBSncores*self.PBSnnodes)) + " ./P-Gadget3 ./param.txt 1>" + self.outfile + " 2>" + self.errorfile - self.SLURMexecute = "mpirun -np " + str(int(self.SLURMcores)) + " ./P-Gadget3 ./param.txt 1>" + self.outfile + " 2>" + self.errorfile - - def _outfile_changed(self): - if self.ENABLE_SUBFIND == True: - self.PBSexecute = "mpirun -np " + str(int(self.PBSncores*self.PBSnnodes)) + " ./P-Gadget3_sub ./param_sub.txt 3 63 1>" + self.outfile + "sub 2>" + self.errorfile + "sub" - self.SLURMexecute = "mpirun -np " + str(int(self.SLURMcores)) + " ./P-Gadget3_sub ./param_sub.txt 3 63 1>" + self.outfile + "sub 2>" + self.errorfile + "sub" - else: - self.PBSexecute = "mpirun -np " + str(int(self.PBSncores*self.PBSnnodes)) + " ./P-Gadget3 ./param.txt 1>" + self.outfile + " 2>" + self.errorfile - self.SLURMexecute = "mpirun -np " + str(int(self.SLURMcores)) + " ./P-Gadget3 ./param.txt 1>" + self.outfile + " 2>" + self.errorfile - - + def _refresh_softenings(self): + """Mirror the derived softening lengths onto the displayed traits.""" + for name, value in gadgetwriter.softenings(self.BoxSize, self.lmax).items(): + setattr(self, "Softening" + name, value) + setattr(self, "Softening" + name + "MaxPhys", value) + + def execute_command(self, scheduler=None): + """The mpirun line for the given scheduler, or for the selected one.""" + scheduler = (scheduler or self.subtype).upper() + ranks = (int(self.SLURMcores) if scheduler == "SLURM" + else int(self.PBSncores) * int(self.PBSnnodes)) + return submit.mpirun_command( + ranks=ranks, + include_subfind=self.ENABLE_SUBFIND, + out=self.outfile, + err=self.errorfile) + + def _refresh_execute(self): + """Keep both displayed execute lines in step with the current settings.""" + self.PBSexecute = self.execute_command("PBS") + self.SLURMexecute = self.execute_command("SLURM") + + # Any of these invalidate both execute lines. + _ENABLE_SUBFIND_changed = _refresh_execute + _PBSncores_changed = _refresh_execute + _PBSnnodes_changed = _refresh_execute + _SLURMcores_changed = _refresh_execute + _errorfile_changed = _refresh_execute + _outfile_changed = _refresh_execute def _vizexpz_button_fired(self): figure = self.main.display @@ -514,255 +484,85 @@ def _redshifti_changed(self): self.expfacti = 1./(1 + self.redshifti) def _BoxSize_changed(self): - self.SofteningGas = 0.0 - self.SofteningHalo = self.BoxSize/2**float(self.lmax)/40. - self.SofteningDisk = 2*self.SofteningHalo - self.SofteningBulge = 2*self.SofteningDisk - self.SofteningStars = 2*self.SofteningBulge - self.SofteningBndry = 2*self.SofteningStars - self.SofteningGasMaxPhys = 0.0 - self.SofteningHaloMaxPhys = self.BoxSize/2**float(self.lmax)/40. - self.SofteningDiskMaxPhys = 2*self.SofteningHaloMaxPhys - self.SofteningBulgeMaxPhys = 2*self.SofteningDiskMaxPhys - self.SofteningStarsMaxPhys = 2*self.SofteningBulgeMaxPhys - self.SofteningBndryMaxPhys = 2*self.SofteningStarsMaxPhys + self._refresh_softenings() def _lmax_changed(self): - if int(self.lmax) <= 11: - self.PMGRID = 256 - elif 12 <= int(self.lmax) <= 13: - self.PMGRID = 512 - elif int(self.lmax) > 13: - self.PMGRID = 1024 - - self.SofteningGas = 0.0 - self.SofteningHalo = self.BoxSize/2**float(self.lmax)/40. - self.SofteningDisk = 2*self.SofteningHalo - self.SofteningBulge = 2*self.SofteningDisk - self.SofteningStars = 2*self.SofteningBulge - self.SofteningBndry = 2*self.SofteningStars - self.SofteningGasMaxPhys = 0.0 - self.SofteningHaloMaxPhys = self.BoxSize/2**float(self.lmax)/40. - self.SofteningDiskMaxPhys = 2*self.SofteningHaloMaxPhys - self.SofteningBulgeMaxPhys = 2*self.SofteningDiskMaxPhys - self.SofteningStarsMaxPhys = 2*self.SofteningBulgeMaxPhys - self.SofteningBndryMaxPhys = 2*self.SofteningStarsMaxPhys - - def constructconfigsh(self,filename): - f = open(filename,'w') - f.write("#!/bin/bash\n") - f.write("##################################################\n") - f.write("# Enable/Disable compile-time options as needed #\n") - f.write("##################################################\n") - f.write("#--------------------------------------- Basic operation mode of code\n") - f.write("PERIODIC\n") - f.write("#COOLING\n") - f.write("#SFR\n") - f.write("#SINKS\n") - f.write("UNEQUALSOFTENINGS\n") - f.write("#NUM_THREADS=4 # Now OpenMP works the same, so don't compile with OpenMP *and* PTHREADS !\n") - f.write("#--------------------------------------- Kernel Options\n") - f.write("#QUINTIC_KERNEL # Implementation of the Morris 1996 quintic spline kernel, requires (3/2)^3 more neighbours !\n") - f.write("#TWODIMS # Switch for 2D test problems\n") - f.write("#ONEDIM # Switch for 1D test problems\n") - f.write("#--------------------------------------- TreePM Options\n") - f.write("PMGRID=" + str(self.PMGRID) + "\n") - f.write("GRIDBOOST=2\n") - f.write("##ASMTH=1.25\n") - f.write("#RCUT=5.25\n") - f.write("PLACEHIGHRESREGION=2\n") - f.write("ENLARGEREGION=1.2\n") - f.write("#--------------------------------------- Multi-Domain and Top-Level Tree options\n") - f.write("MULTIPLEDOMAINS=8\n") - f.write("#TOPNODEFACTOR=3.0\n") - f.write("#KD_HMAX_ESTIMATE # Alternative way to update HMAX within Tree nodes\n") - f.write("#--------------------------------------- Things that are always recommended\n") - f.write("PEANOHILBERT\n") - f.write("WALLCLOCK\n") - f.write("MYSORT\n") - f.write("#AUTO_SWAP_ENDIAN_READIC # Enables automatic ENDIAN swapping for reading ICs\n") - f.write("#WRITE_KEY_FILES # Enables writing key index files\n") - f.write("#WRITE_INFO_BLOCK # Enables writing the INFO block\n") - f.write("#PERMUTATAION_OPTIMIZATION\n") - f.write("#PROCESS_TIMES_OF_OUTPUTLIST # Chooses the outputtime closest to any global step\n") - f.write("#SYNCRONIZ_OUTPUT # Writes output only at global time steps\n") - f.write("#---------------------------------------- Single/Double Precision\n") - f.write("DOUBLEPRECISION\n") - f.write("DOUBLEPRECISION_FFTW\n") - f.write("#OUTPUT_IN_DOUBLEPRECISION # snapshot files will be written in double precision\n") - f.write("#INPUT_IN_DOUBLEPRECISION\n") - f.write("#---------------------------------------- Invariance Test\n") - f.write("#INVARIANCETEST\n") - f.write("#INVARIANCETEST_SIZE1=2\n") - f.write("#INVARIANCETEST_SIZE2=6\n") - f.write("#FLTROUNDOFFREDUCTION # enables (expensive!) `double-double' round-off reduction in particle sums\n") - f.write("#SOFTDOUBLEDOUBLE # needs to be set if a C++ software implementation of 128bit double-double precision should be used\n") - f.write("#---------------------------------------- On the fly FOF groupfinder\n") - - if self.ENABLE_SUBFIND == True: - f.write("FOF\n") - f.write("FOF_PRIMARY_LINK_TYPES=2 # 2^type for the primary dark matter type\n") - f.write("FOF_SECONDARY_LINK_TYPES=" + str(self.FOF_SECONDARY_LINK_TYPES) + " # 2^type for the types linked to nearest primaries\n") - else: - f.write("#FOF\n") - f.write("#FOF_PRIMARY_LINK_TYPES=2 # 2^type for the primary dark matter type\n") - f.write("#FOF_SECONDARY_LINK_TYPES=" + str(self.FOF_SECONDARY_LINK_TYPES) + " # 2^type for the types linked to nearest primaries\n") - - f.write("#FOF_GROUP_MIN_LEN=32 # default is 32\n") - - if self.ENABLE_SUBFIND == True: - f.write("SUBFIND\n") - else: - f.write("#SUBFIND\n") - - f.write("#DENSITY_SPLIT_BY_TYPE=1+2+16+32 # 2^type for whch the densities should be calculated seperately\n") - f.write("#MAX_NGB_CHECK=3 # Max numbers of neighbours for sattlepoint detection (default = 2)\n") - f.write("#SAVE_MASS_TAB # Saves the an additional array with the masses of the different components\n") - f.write("#SUBFINDSAVE_PARTICLELISTS # Saves also phase-space and type variables parallel to IDs\n") - f.write("#SO_VEL_DISPERSIONS # computes velocity dispersions for as part of FOF SO-properties\n") - f.write("#ORDER_SNAPSHOTS_BY_ID\n") - f.write("#SAVE_HSML_IN_IC_ORDER # will store the hsml-values in the order of the particles in the IC file\n") - f.write("#ONLY_PRODUCE_HSML_FILES # only carries out density estimate\n") - f.write("#KEEP_HSML_AS_GUESS # keep using hsml for gas particles in subfind_density\n") - f.write("#LINKLENGTH=0.16 # Linkinglength for FoF (default=0.2)\n") - f.write("#NO_GAS_CLOUDS # Do not accept pure gaseous substructures\n") - f.write("#WRITE_SUB_IN_SNAP_FORMAT # Save subfind results in snap format\n") - f.write("#LT_ADD_GAL_TO_SUB=12 # Adds optical luminosities in 6 bands to subhalos\n") - f.write("#DUSTATT=11 # Includes dust attenuation into the luminosity calculation (using 11 radial bins)\n") - f.write("#OBSERVER_FRAME # If defined, use CB07 Observer Frame Luminosities, otherwise CB07 Rest Frame Luminosities\n") - f.write("#SO_BAR_INFO # Adds temperature, Lx, bfrac, etc to Groups\n") - f.write("#FSUBFINDCOUNT_BIG_HALOS=1e4 # Adds extra blocks for Halos with M_TopHat > SUBFIND_COUNT_BIG_HALOS\n") - f.write("#KD_CHOOSE_PSUBFIND_LIMIT # Increases the limit for the parallel subfind to the maximum possible\n") - f.write("#KD_ALTERNATIVE_GROUP_SORT # Alternative way to sort the Groups/SubGroupe before writing\n") - f.write("#KD_CHOOSE_LINKING_LENGTH # Special way to estimate the linking length\n") - f.write("#SUBFINDREAD_FOF\n") - f.write("#SUBFINDCOLLECTIVE_STAGE1\n") - f.write("#SUBFINDCOLLECTIVE_STAGE2\n") - f.write("#SUBFINDALTERNATIVE_COLLECTIVE\n") - f.write("#SUBFINDRESHUFFLE_CATALOGUE\n") - f.write("#SUBFINDRESHUFFLE_CATALOGUE_WITH_VORONOI\n") - f.write("#SUBFINDRESHUFFLE_AND_POTENTIAL #needs -DSUBFIND_RESHUFFLE_CATALOGUE and COMPUTE_POTENTIAL_ENERGY\n") - f.write("#SUBFINDDENSITY_AND_POTENTIAL #only calculated density and potential and write them into snapshot\n") - f.close() - - def makeparam(self,filename,ext,includesub): - snapshots = range(0,int(self.nintervals)) - expfact = np.linspace(self.expfacti,1,int(self.nintervals)) - expfilename = ext + '/' + self.OutputListFilename - f = open(expfilename,'w') - - for i in range(0,len(self.ExpansionListArr[:,0])): - f.write(str(np.float64(self.ExpansionListArr[i,2])) + ' ' + str(int(self.ExpansionListArr[i,3])) + '\n') - - f.close() - - f = open(filename,'w') - f.write('%---- Relevant files' + '\n') - f.write('InitCondFile ' + str(self.InitCondFile) + '\n') - f.write('OutputDir ' + str(self.OutputDir) + '\n') - f.write('EnergyFile ' + str(self.EnergyFile) + '\n') - f.write('InfoFile ' + str(self.InfoFile) + '\n') - f.write('TimingsFile ' + str(self.TimingsFile) + '\n') - f.write('CpuFile ' + str(self.CpuFile) + '\n') - f.write('RestartFile ' + str(self.RestartFile) + '\n') - f.write('SnapshotFileBase ' + str(self.SnapshotFileBase) + '\n') - f.write('OutputListFilename ' + str(self.OutputListFilename) + '\n') - f.write('TimebinFile ' + str(self.TimebinFile) + '\n') - f.write('\n') - f.write('%---- File formats' + '\n') - f.write('ICFormat ' + str(self.ICFormat) + '\n') - f.write('SnapFormat ' + str(self.SnapFormat) + '\n') - f.write('\n') - f.write('%---- CPU-time limits' + '\n') - f.write('TimeLimitCPU ' + str(self.TimeLimitCPU) + '\n') - f.write('CpuTimeBetRestartFile ' + str(self.CpuTimeBetRestartFile) + '\n') - f.write('ResubmitOn ' + str(self.ResubmitOn) + '\n') - f.write('ResubmitCommand ' + str(self.ResubmitCommand) + '\n') - f.write('\n') - f.write('%----- Memory alloction' + '\n') - f.write('MaxMemSize ' + str(self.MaxMemSize) + '\n') - f.write('PartAllocFactor ' + str(self.PartAllocFactor) + '\n') - f.write('BufferSize ' + str(self.BufferSize) + '\n') - f.write('\n') - f.write('%---- Caracteristics of run' + '\n') - f.write('TimeBegin ' + str(self.TimeBegin) + '\n') - f.write('TimeMax ' + str(self.TimeMax) + '\n') - f.write('\n') - f.write('%---- Basic code options that set the type of simulation' + '\n') - f.write('ComovingIntegrationOn ' + str(self.ComovingIntegrationOn) + '\n') - f.write('PeriodicBoundariesOn ' + str(self.PeriodicBoundariesOn) + '\n') - f.write('CoolingOn ' + str(self.CoolingOn) + '\n') - f.write('StarformationOn ' + str(self.StarformationOn) + '\n') - f.write('\n') - f.write('%---- Cosmological parameters' + '\n') - f.write('Omega0 ' + str(self.Omega0) + '\n') - f.write('OmegaLambda ' + str(self.OmegaLambda) + '\n') - f.write('OmegaBaryon ' + str(self.OmegaBaryon) + '\n') - f.write('HubbleParam ' + str(self.HubbleParam/100) + '\n') - f.write('BoxSize ' + str(self.BoxSize) + '\n') - f.write('\n') - f.write('%---- Tree algorithm, force accuracy, domain update frequency' + '\n') - f.write('OutputListOn ' + str(self.OutputListOn) + '\n') - f.write('TimeBetSnapshot ' + str(self.TimeBetSnapshot) + '\n') - f.write('TimeOfFirstSnapshot ' + str(self.TimeOfFirstSnapshot) + '\n') - f.write('TimeBetStatistics ' + str(self.TimeBetStatistics) + '\n') - f.write('NumFilesPerSnapshot ' + str(self.NumFilesPerSnapshot) + '\n') - f.write('NumFilesWrittenInParallel ' + str(self.NumFilesWrittenInParallel) + '\n') - f.write('\n') - f.write('%---- Accuracy of time integration' + '\n') - f.write('TypeOfTimestepCriterion ' + str(self.TypeOfTimestepCriterion) + '\n') - f.write('ErrTolIntAccuracy ' + str(self.ErrTolIntAccuracy) + '\n') - f.write('CourantFac ' + str(self.CourantFac) + '\n') - f.write('MaxRMSDisplacementFac ' + str(self.MaxRMSDisplacementFac) + '\n') - f.write('MaxSizeTimestep ' + str(self.MaxSizeTimestep) + '\n') - f.write('MinSizeTimestep ' + str(self.MinSizeTimestep) + '\n') - f.write('\n') - #f.write('%---- Accuracy of time integration') - f.write('InitGasTemp ' + str(self.InitGasTemp) + '\n') - f.write('MinGasTemp ' + str(self.MinGasTemp) + '\n') - f.write('\n') - f.write('%---- Tree algorithm, force accuracy, domain update frequency' + '\n') - f.write('TypeOfOpeningCriterion ' + str(self.TypeOfOpeningCriterion) + '\n') - f.write('ErrTolTheta ' + str(self.ErrTolTheta) + '\n') - f.write('ErrTolForceAcc ' + str(self.ErrTolForceAcc) + '\n') - f.write('TreeDomainUpdateFrequency ' + str(self.TreeDomainUpdateFrequency) + '\n') - f.write('\n') - f.write('%---- Initial density estimate' + '\n') - f.write('DesNumNgb ' + str(self.DesNumNgb) + '\n') - f.write('MaxNumNgbDeviation ' + str(self.MaxNumNgbDeviation) + '\n') - f.write('\n') - f.write('%---- System of units' + '\n') - f.write('UnitLength_in_cm ' + str(self.UnitLength_in_cm) + '\n') - f.write('UnitMass_in_g ' + str(self.UnitMass_in_g) + '\n') - f.write('UnitVelocity_in_cm_per_s ' + str(self.UnitVelocity_in_cm_per_s) + '\n') - f.write('GravityConstantInternal ' + str(self.GravityConstantInternal) + '\n') - f.write('\n') - f.write('%---- Gravitational softening lengths' + '\n') - f.write('MinGasHsmlFractional ' + str(self.MinGasHsmlFractional) + '\n') - f.write('\n') - f.write('SofteningGas ' + str(self.SofteningGas) + '\n') - f.write('SofteningHalo ' + str(self.SofteningHalo) + '\n') - f.write('SofteningDisk ' + str(self.SofteningDisk) + '\n') - f.write('SofteningBulge ' + str(self.SofteningBulge) + '\n') - f.write('SofteningStars ' + str(self.SofteningStars) + '\n') - f.write('SofteningBndry ' + str(self.SofteningBndry) + '\n') - f.write('\n') - f.write('SofteningGasMaxPhys ' + str(self.SofteningGasMaxPhys) + '\n') - f.write('SofteningHaloMaxPhys ' + str(self.SofteningHaloMaxPhys) + '\n') - f.write('SofteningDiskMaxPhys ' + str(self.SofteningDiskMaxPhys) + '\n') - f.write('SofteningBulgeMaxPhys ' + str(self.SofteningBulgeMaxPhys) + '\n') - f.write('SofteningStarsMaxPhys ' + str(self.SofteningStarsMaxPhys) + '\n') - f.write('SofteningBndryMaxPhys ' + str(self.SofteningBndryMaxPhys) + '\n') - f.write('\n') - f.write('%---- non-common' + '\n') - f.write('ArtBulkViscConst ' + str(self.ArtBulkViscConst) + '\n') - - if includesub == True: - f.write('ErrTolThetaSubfind ' + str(self.ErrTolThetaSubfind) + '\n') - f.write('DesLinkNgb ' + str(self.DesLinkNgb) + '\n') - - f.close() - + self.PMGRID = gadgetwriter.pmgrid_for_levelmax(self.lmax) + self._refresh_softenings() + + def write_config_sh(self, filename): + """Emit a P-Gadget3 Config.sh for the current compile-time settings.""" + return writers.write(filename, gadgetwriter.config_sh( + pmgrid=self.PMGRID, + include_subfind=self.ENABLE_SUBFIND, + fof_secondary_link_types=self.FOF_SECONDARY_LINK_TYPES)) + + def makeparam(self, filename, ext, includesub): + """Emit a Gadget parameter file plus its ExpansionList, into ``ext``.""" + writers.write( + os.path.join(ext, self.OutputListFilename), + gadgetwriter.expansion_list( + (row[2], row[3]) for row in self.ExpansionListArr)) + + return writers.write(filename, gadgetwriter.param_file( + cosmo=cosmology.get(self.cosmologylist[0]), + boxsize=self.BoxSize, + levelmax=self.lmax, + init_cond_file=self.InitCondFile, + output_dir=self.OutputDir, + energy_file=self.EnergyFile, + info_file=self.InfoFile, + timings_file=self.TimingsFile, + cpu_file=self.CpuFile, + restart_file=self.RestartFile, + snapshot_file_base=self.SnapshotFileBase, + output_list_filename=self.OutputListFilename, + timebin_file=self.TimebinFile, + ic_format=self.ICFormat, + snap_format=self.SnapFormat, + time_limit_cpu=self.TimeLimitCPU, + cpu_time_bet_restart_file=self.CpuTimeBetRestartFile, + resubmit_on=self.ResubmitOn, + resubmit_command=self.ResubmitCommand, + max_mem_size=self.MaxMemSize, + part_alloc_factor=self.PartAllocFactor, + buffer_size=self.BufferSize, + time_begin=self.TimeBegin, + time_max=self.TimeMax, + comoving_integration_on=self.ComovingIntegrationOn, + periodic_boundaries_on=self.PeriodicBoundariesOn, + cooling_on=self.CoolingOn, + star_formation_on=self.StarformationOn, + omega_baryon=self.OmegaBaryon, + output_list_on=self.OutputListOn, + time_bet_snapshot=self.TimeBetSnapshot, + time_of_first_snapshot=self.TimeOfFirstSnapshot, + time_bet_statistics=self.TimeBetStatistics, + num_files_per_snapshot=self.NumFilesPerSnapshot, + num_files_written_in_parallel=self.NumFilesWrittenInParallel, + type_of_timestep_criterion=self.TypeOfTimestepCriterion, + err_tol_int_accuracy=self.ErrTolIntAccuracy, + courant_fac=self.CourantFac, + max_rms_displacement_fac=self.MaxRMSDisplacementFac, + max_size_timestep=self.MaxSizeTimestep, + min_size_timestep=self.MinSizeTimestep, + init_gas_temp=self.InitGasTemp, + min_gas_temp=self.MinGasTemp, + type_of_opening_criterion=self.TypeOfOpeningCriterion, + err_tol_theta=self.ErrTolTheta, + err_tol_force_acc=self.ErrTolForceAcc, + tree_domain_update_frequency=self.TreeDomainUpdateFrequency, + des_num_ngb=self.DesNumNgb, + max_num_ngb_deviation=self.MaxNumNgbDeviation, + unit_length_in_cm=self.UnitLength_in_cm, + unit_mass_in_g=self.UnitMass_in_g, + unit_velocity_in_cm_per_s=self.UnitVelocity_in_cm_per_s, + gravity_constant_internal=self.GravityConstantInternal, + min_gas_hsml_fractional=self.MinGasHsmlFractional, + art_bulk_visc_const=self.ArtBulkViscConst, + include_subfind=includesub, + err_tol_theta_subfind=self.ErrTolThetaSubfind, + des_link_ngb=self.DesLinkNgb)) def _checkexistence_button_fired(self): @@ -912,8 +712,7 @@ def __init__(self, main, **kwargs): self.FOF_SECONDARY_LINK_TYPES = '4+8+16+32' self.ExpansionListArr = np.zeros(shape=(1,4)) self.clusteropt = self.main.headertab.clusteropt - self.PBSexecute = "mpirun -np " + str(int(self.PBSncores*self.PBSnnodes)) + " ./P-Gadget3 ./param.txt 1>" + self.outfile + " 2>" + self.errorfile - self.SLURMexecute = "mpirun -np " + str(int(self.SLURMcores)) + " ./P-Gadget3 ./param.txt 1>" + self.outfile + " 2>" + self.errorfile + self._refresh_execute() #self.PBSstring = \ #"#!/bin/sh \n#PBS -l nodes=3:ppn=8 \n#PBS -N H190897LX9N1 \n#PBS -m be \n. /opt/torque/etc/openmpi-setup.sh \ncd /bigbang/data/AnnaGroup/caterpillar/halos/H190897/H190897_BE_Z127_P7_LN7_LX9_O4_NV1 \nmpirun -np 24 ./P-Gadget3 ./param.txt 1>OUTPUT 2>ERROR" diff --git a/src/cme/ui/ics.py b/src/cme/ui/ics.py index 87774ac..108013c 100644 --- a/src/cme/ui/ics.py +++ b/src/cme/ui/ics.py @@ -1,6 +1,10 @@ from cme.ui.common import * -from cme import cosmology, naming +from cme import cosmology, naming, writers +from cme.writers import music + +#: Side length of the Caterpillar parent box, in Mpc/h. +PARENT_BOXLENGTH = 100 from glob import glob import random @@ -232,25 +236,37 @@ def _generate_button_fired(self): os.makedirs(self.outpath + 'parent/' + foldername) writepath = self.outpath + 'parent/' + foldername - confname = self.outpath + 'parent/' + foldername + '/' + foldername + '.conf' - - - omegam,omegal,omegab,hubble,sigma8,nspec = cosmology.cosmoconstant(cosmi) - - baryonsstr = determineboolstr(self.baryons) - use2LPTstr = determineboolstr(self.use2LPT) - useLLAstr = determineboolstr(self.useLLA) - periodicTFstr = determineboolstr(self.periodicTF) - fftfinestr = determineboolstr(self.fftfine) - alignstr = determineboolstr(self.align) - padding = 8 - overlap = 4 - - constructparentconf(confname,boxlengthi,self.zinit,self.lTF,padding,overlap,self.refx,self.refy,self.refz, \ - self.extentx,self.extenty,self.extentz,alignstr,baryonsstr,use2LPTstr,useLLAstr,omegam,omegal,omegab,hubble, \ - sigma8,nspec,self.tranfunc[0],self.seedinit,self.outformat,'./' + self.outfilename,fftfinestr,self.accuracy,self.presmooth,self.postsmooth, \ - self.smoother,self.laplaceorder,self.gradorder,self.boxlevel,periodicTFstr,self.noutput) - + confname = os.path.join(writepath, foldername + '.conf') + + writers.write(confname, music.parent_conf( + cosmo=cosmology.get(cosmi), + boxlength=boxlengthi, + zstart=self.zinit, + level=self.boxlevel, + levelmin_TF=self.lTF, + padding=8, + overlap=4, + ref_center=(self.refx, self.refy, self.refz), + ref_extent=(self.extentx, self.extenty, self.extentz), + seed=self.seedinit, + align_top=self.align, + baryons=self.baryons, + use_2LPT=self.use2LPT, + use_2LLA=self.useLLA, + periodic_TF=self.periodicTF, + transfer=self.tranfunc[0], + out_format=self.outformat, + ic_filename='./' + self.outfilename, + num_files=self.noutput, + fft_fine=self.fftfine, + accuracy=self.accuracy, + pre_smooth=self.presmooth, + post_smooth=self.postsmooth, + smoother=self.smoother, + laplace_order=self.laplaceorder, + grad_order=self.gradorder, + )) + self.confstatus = 'Generated halo configuration files.' runmusic = self.musicpath + '/MUSIC ' + confname cding = "cd " + writepath @@ -299,21 +315,41 @@ def _generate_button_fired(self): #print self.parentbox #print self.centx,self.centy,self.centz #print self.extx,self.exty,self.extz - omegam,omegal,omegab,hubble,sigma8,nspec = cosmology.cosmoconstant(cosmi) - - baryonsstr = determineboolstr(self.baryons) - use2LPTstr = determineboolstr(self.use2LPT) - useLLAstr = determineboolstr(self.useLLA) - periodicTFstr = determineboolstr(self.periodicTF) - fftfinestr = determineboolstr(self.fftfine) - alignstr = determineboolstr(self.align) - boxlength = 100 - #print self.haloidselect - constructresimconf(confname,boxlength,self.zinit,lmini,self.lTF,lmaxi,paddingi,overlapi,self.centx,self.centy,self.centz, \ - self.extx,self.exty,self.extz,alignstr,baryonsstr,use2LPTstr,useLLAstr,omegam,omegal,omegab,hubble, \ - sigma8,nspec,self.tranfunc[0],self.parentseednum,self.parentseedlevel,self.outformat,'./' + self.outfilename,fftfinestr,self.accuracy,self.presmooth,self.postsmooth, \ - self.smoother,self.laplaceorder,self.gradorder,self.boxlevel,periodicTFstr,pointfile,boxtypei,self.noutput,self.haloidselect,nrviri) - + writers.write(confname, music.resim_conf( + cosmo=cosmology.get(cosmi), + boxlength=PARENT_BOXLENGTH, + zstart=self.zinit, + levelmin=lmini, + levelmin_TF=self.lTF, + levelmax=lmaxi, + padding=paddingi, + overlap=overlapi, + region=boxtypei, + region_point_file=pointfile, + ref_center=(self.centx, self.centy, self.centz), + ref_extent=(self.extx, self.exty, self.extz), + seed_level=self.parentseedlevel, + base_seed=self.parentseednum, + halo_id=self.haloidselect, + nrvir=nrviri, + align_top=self.align, + baryons=self.baryons, + use_2LPT=self.use2LPT, + use_2LLA=self.useLLA, + periodic_TF=self.periodicTF, + transfer=self.tranfunc[0], + out_format=self.outformat, + ic_filename='./' + self.outfilename, + num_files=self.noutput, + fft_fine=self.fftfine, + accuracy=self.accuracy, + pre_smooth=self.presmooth, + post_smooth=self.postsmooth, + smoother=self.smoother, + laplace_order=self.laplaceorder, + grad_order=self.gradorder, + )) + self.confstatus = 'Generated halo configuration files.' if self.executemusic == True: runmusic = self.musicpath + '/MUSIC ' + confname @@ -735,163 +771,3 @@ def __init__(self, main, **kwargs): self.parentconfstatus = "Proceed" except IOError: self.parentconfstatus = "Cannot locate file." - -def constructresimconf(confname,boxlength,zstart,lmin,lTF,lmax,padding,overlap,refcentx,refcenty,refcentz, \ - refextx,refexty,refextz,align,baryons,use2LPT,useLLA,omegam,omegal,omegab,hubble, \ - sigma8,nspec,transfer,seednum,seedlevel,outformat,icfilename,fftfine,accuracy,presmooth,postsmooth, \ - smoother,laplaceorder,gradorder,boxlevel,periodicTFstr,pointfile,boxtype,noutput,haloid,nrvir): - - f = open(confname,'w') - f.write('[setup]' + '\n') - f.write('boxlength = ' + str(boxlength) + '\n') - f.write('zstart = ' + str(zstart) + '\n') - f.write('levelmin = ' + str(lmin) + '\n') - f.write('levelmin_TF = ' + str(lTF) + '\n') - f.write('levelmax = ' + str(lmax) + '\n') - f.write('padding = ' + str(padding) + '\n') - f.write('overlap = ' + str(overlap) + '\n') - f.write('region = ' + str(boxtype) + '\n') - - if boxtype == 'box': - f.write('ref_center = ' + str(refcentx) + ',' + str(refcenty) + ',' + str(refcentz) + '\n') - f.write('ref_extent = ' + str(refextx) + ',' + str(refexty) + ',' + str(refextz) + '\n') - - f.write('region_point_file = ' + str(pointfile) + '\n') - f.write('align_top = ' + str(align) + '\n') - f.write('baryons = ' + str(baryons) + '\n') - f.write('use_2LPT = ' + str(use2LPT) + '\n') - f.write('use_2LLA = ' + str(useLLA) + '\n') - f.write('periodic_TF = ' + str(periodicTFstr) + '\n') - f.write('\n') - f.write('[cosmology]'+ '\n') - f.write('Omega_m = ' + str(omegam) + '\n') - f.write('Omega_L = ' + str(omegal) + '\n') - f.write('Omega_b = ' + str(omegab) + '\n') - f.write('H0 = ' + str(hubble) + '\n') - f.write('sigma_8 = ' + str(sigma8) + '\n') - f.write('nspec = ' + str(nspec) + '\n') - f.write('transfer = ' + str(transfer) + '\n') - f.write('\n') - f.write('[random]' + '\n') - - #diff = int(lmax)+1 - int(lmin) - #print diff - #print int(lmax),int(lmin) - #seednumnew = random.sample(range(10000,99999), diff) - - #seedi = 0 - #print haloid - #mult = nrvir - #f.write('seed[' + str(7) + '] = ' + str(345) + '\n') - #f.write('seed[' + str(8) + '] = ' + str(3456) + '\n') - #f.write('seed[' + str(9) + '] = ' + str(34567) + '\n') - delta = int(nrvir) - for level in range(seedlevel,int(lmax)+1): - delta += delta*2 + 1 - if level != seedlevel: - seeduse = int(haloid) + int(delta) - #seeduse = seednumnew[seedi] - f.write('seed[' + str(level) + '] = ' + str(seeduse) + '\n') - elif level == seedlevel: - f.write('seed[' + str(seedlevel) + '] = ' + str(seednum) + '\n') - #seedi += 1 - - #f.write('seed[7] = ' + str(seeduse) + '\n') - #f.write('seed[8] = ' + str(seeduse) + '\n') - #f.write('seed[9] =' str(34567\n') - #f.write('seed[10] = ' + str(seeduse) + '\n') - #f.write('seed[11] = ' + str(seeduse) + '\n') - - #if int(lmax) == 12: - # f.write('seed[12] = 2600\n') - - - - f.write('\n') - f.write('[output]' + '\n') - - if outformat == 'music': - outformat = 'generic' - - if outformat == 'gadget': - outformat = 'gadget2' - - f.write('format = ' + str(outformat) + '\n') - f.write('filename = ' + str(icfilename) + '\n') - f.write('gadget_num_files = ' + str(noutput) + '\n') - f.write('\n') - f.write('[poisson]' + '\n') - f.write('fft_fine = ' + str(fftfine) + '\n') - f.write('accuracy = ' + str(accuracy) + '\n') - f.write('pre_smooth = ' + str(presmooth) + '\n') - f.write('post_smooth = ' + str(postsmooth) + '\n') - f.write('smoother = ' + str(smoother) + '\n') - f.write('laplace_order = ' + str(laplaceorder) + '\n') - f.write('grad_order = ' + str(gradorder) + '\n') - f.close() - -def constructparentconf(confname,boxlength,zinit,lTF,padding,overlap,refcentx,refcenty,refcentz, \ - refextx,refexty,refextz,align,baryons,use2LPT,useLLA,omegam,omegal,omegab,hubble, \ - sigma8,nspec,transfer,seed,outformat,icfilename,fftfine,accuracy,presmooth,postsmooth, \ - smoother,laplaceorder,gradorder,boxlevel,periodicTFstr,noutput): - - f = open(confname,'w') - f.write('[setup]' + '\n') - f.write('boxlength = ' + str(boxlength) + '\n') - f.write('zstart = ' + str(zinit) + '\n') - f.write('levelmin = ' + str(boxlevel) + '\n') - f.write('levelmin_TF = ' + str(lTF) + '\n') - f.write('levelmax = ' + str(boxlevel) + '\n') - f.write('padding = ' + str(padding) + '\n') - f.write('overlap = ' + str(overlap) + '\n') - f.write('ref_center = ' + str(refcentx) + ', ' + str(refcenty) + ', ' + str(refcentz) + '\n') - f.write('ref_extent = ' + str(refextx) + ', ' + str(refexty) + ', ' + str(refextz) + '\n') - f.write('align_top = ' + str(align) + '\n') - f.write('baryons = ' + str(baryons) + '\n') - f.write('use_2LPT = ' + str(use2LPT) + '\n') - f.write('use_2LLA = ' + str(useLLA) + '\n') - f.write('periodic_TF = ' + str(periodicTFstr) + '\n') - f.write('\n') - f.write('[cosmology]'+ '\n') - f.write('Omega_m = ' + str(omegam) + '\n') - f.write('Omega_L = ' + str(omegal) + '\n') - f.write('Omega_b = ' + str(omegab) + '\n') - f.write('H0 = ' + str(hubble) + '\n') - f.write('sigma_8 = ' + str(sigma8) + '\n') - f.write('nspec = ' + str(nspec) + '\n') - f.write('transfer = ' + str(transfer) + '\n') - f.write('\n') - f.write('[random]' + '\n') - f.write('seed[' + str(boxlevel) + '] = ' + str(seed) + '\n') - - f.write('\n') - f.write('[output]' + '\n') - - if outformat == 'music': - outformat = 'generic' - - if outformat == 'gadget': - outformat = 'gadget2' - - f.write('format = ' + str(outformat) + '\n') - f.write('filename = ' + str(icfilename) + '\n') - f.write('gadget_num_files = ' + str(noutput) + '\n') - f.write('\n') - f.write('[poisson]' + '\n') - f.write('fft_fine = ' + str(fftfine) + '\n') - f.write('accuracy = ' + str(accuracy) + '\n') - f.write('pre_smooth = ' + str(presmooth) + '\n') - f.write('post_smooth = ' + str(postsmooth) + '\n') - f.write('smoother = ' + str(smoother) + '\n') - f.write('laplace_order = ' + str(laplaceorder) + '\n') - f.write('grad_order = ' + str(gradorder) + '\n') - f.close() - -def determineboolstr(boolean): - if boolean == True: - returnstr = 'yes' - elif boolean == False: - returnstr = 'no' - - return returnstr - diff --git a/src/cme/writers/__init__.py b/src/cme/writers/__init__.py new file mode 100644 index 0000000..e1f5d0b --- /dev/null +++ b/src/cme/writers/__init__.py @@ -0,0 +1,22 @@ +"""Pure text emitters for the configuration files this tool generates. + +Every function here takes plain scalars and returns a string. Nothing touches +the filesystem, the GUI, or global state, so the exact bytes handed to MUSIC, +Gadget and the batch scheduler can be asserted in tests -- which is the only +practical defence against a silently malformed parameter file consuming a +cluster allocation before anyone notices. + +Use :func:`write` when the text needs to land on disk. +""" + +import os + + +def write(path, text): + """Write ``text`` to ``path``, creating parent directories as needed.""" + parent = os.path.dirname(os.path.abspath(path)) + if parent: + os.makedirs(parent, exist_ok=True) + with open(path, "w") as handle: + handle.write(text) + return path diff --git a/src/cme/writers/gadget.py b/src/cme/writers/gadget.py new file mode 100644 index 0000000..a569a2c --- /dev/null +++ b/src/cme/writers/gadget.py @@ -0,0 +1,443 @@ +"""P-Gadget3 parameter file, Config.sh and output-list generation. + +Extracted from ``makeparam`` and ``constructconfigsh`` on the Gadget tab, +which built these files with ~110 and ~90 sequential ``f.write`` calls +respectively, interleaved with GUI state. + +Gadget's parameter parser is whitespace-separated ``key value``; keys are +left-justified to a fixed column purely so the file stays readable. +""" + +#: Column width parameter names are padded to. +_KEY_WIDTH = 27 + + +def _params(pairs): + return "".join(f"{key:<{_KEY_WIDTH}}{value}\n" for key, value in pairs) + + +def softenings(boxsize, levelmax): + """Gravitational softening lengths, scaled to the finest refinement level. + + The halo softening is 1/40th of the mean interparticle spacing at + ``levelmax``; each heavier particle type doubles the one before it. Gas is + zero because these are dark-matter-only runs. + + Returns a dict keyed by Gadget's particle type names. + + >>> s = softenings(100.0, 11) + >>> s["Halo"] == 100.0 / 2**11 / 40 + True + >>> s["Disk"] == 2 * s["Halo"] + True + """ + halo = float(boxsize) / 2 ** float(levelmax) / 40.0 + disk = 2 * halo + bulge = 2 * disk + stars = 2 * bulge + bndry = 2 * stars + return { + "Gas": 0.0, + "Halo": halo, + "Disk": disk, + "Bulge": bulge, + "Stars": stars, + "Bndry": bndry, + } + + +def pmgrid_for_levelmax(levelmax): + """The particle-mesh grid size appropriate to a refinement level. + + >>> [pmgrid_for_levelmax(n) for n in (10, 11, 12, 13, 14)] + [256, 256, 512, 512, 1024] + """ + levelmax = int(levelmax) + if levelmax <= 11: + return 256 + if levelmax <= 13: + return 512 + return 1024 + + +def expansion_list(rows): + """The ``ExpansionList`` file: one `` `` line. + + ``rows`` is an iterable of ``(expansion_factor, flag)`` pairs. + """ + return "".join(f"{float(a)} {int(flag)}\n" for a, flag in rows) + + +def param_file( + *, + cosmo, + boxsize, + levelmax, + init_cond_file="./ics_rewrite", + output_dir="./outputs", + energy_file="energy.txt", + info_file="info.txt", + timings_file="timings.txt", + cpu_file="cpu.txt", + restart_file="restart", + snapshot_file_base="snap", + output_list_filename="ExpansionList", + timebin_file="timebin", + ic_format=1, + snap_format=1, + time_limit_cpu=340000, + cpu_time_bet_restart_file=43200, + resubmit_on=0, + resubmit_command="my-scriptfile", + max_mem_size=3500, + part_alloc_factor=2.5, + buffer_size=100, + time_begin=0.0078125, + time_max=1.0, + comoving_integration_on=1, + periodic_boundaries_on=1, + cooling_on=0, + star_formation_on=0, + omega_baryon=0.0, + output_list_on=1, + time_bet_snapshot=0.0, + time_of_first_snapshot=0.0, + time_bet_statistics=0.01, + num_files_per_snapshot=4, + num_files_written_in_parallel=4, + type_of_timestep_criterion=0, + err_tol_int_accuracy=0.012, + courant_fac=0.15, + max_rms_displacement_fac=0.125, + max_size_timestep=0.005, + min_size_timestep=0.0, + init_gas_temp=1000.0, + min_gas_temp=5.0, + type_of_opening_criterion=1, + err_tol_theta=0.6, + err_tol_force_acc=0.0025, + tree_domain_update_frequency=0.01, + des_num_ngb=64, + max_num_ngb_deviation=1, + unit_length_in_cm=3.085678e24, + unit_mass_in_g=1.989e43, + unit_velocity_in_cm_per_s=1e5, + gravity_constant_internal=0, + min_gas_hsml_fractional=0.25, + art_bulk_visc_const=1.0, + include_subfind=False, + err_tol_theta_subfind=0.7, + des_link_ngb=20, + softening=None, +): + """Build a P-Gadget3 parameter file. + + ``cosmo`` is a :class:`cme.cosmology.Cosmology`; ``HubbleParam`` is written + as its dimensionless ``h``. The GUI previously stored H0 in a trait named + ``HubbleParam`` and divided by 100 on the way out, so the value shown in + the interface disagreed with the value written to disk. Taking the + cosmology directly removes that conversion. + + ``omega_baryon`` is separate from ``cosmo.omega_b`` because dark-matter-only + runs set it to zero while still using the cosmology's other parameters. + + ``softening`` defaults to :func:`softenings` for ``boxsize``/``levelmax``. + """ + soft = softenings(boxsize, levelmax) if softening is None else softening + + sections = [ + ("%---- Relevant files", [ + ("InitCondFile", init_cond_file), + ("OutputDir", output_dir), + ("EnergyFile", energy_file), + ("InfoFile", info_file), + ("TimingsFile", timings_file), + ("CpuFile", cpu_file), + ("RestartFile", restart_file), + ("SnapshotFileBase", snapshot_file_base), + ("OutputListFilename", output_list_filename), + ("TimebinFile", timebin_file), + ]), + ("%---- File formats", [ + ("ICFormat", ic_format), + ("SnapFormat", snap_format), + ]), + ("%---- CPU-time limits", [ + ("TimeLimitCPU", time_limit_cpu), + ("CpuTimeBetRestartFile", cpu_time_bet_restart_file), + ("ResubmitOn", resubmit_on), + ("ResubmitCommand", resubmit_command), + ]), + ("%----- Memory allocation", [ + ("MaxMemSize", max_mem_size), + ("PartAllocFactor", part_alloc_factor), + ("BufferSize", buffer_size), + ]), + ("%---- Characteristics of run", [ + ("TimeBegin", time_begin), + ("TimeMax", time_max), + ]), + ("%---- Basic code options that set the type of simulation", [ + ("ComovingIntegrationOn", comoving_integration_on), + ("PeriodicBoundariesOn", periodic_boundaries_on), + ("CoolingOn", cooling_on), + ("StarformationOn", star_formation_on), + ]), + ("%---- Cosmological parameters", [ + ("Omega0", cosmo.omega_m), + ("OmegaLambda", cosmo.omega_l), + ("OmegaBaryon", omega_baryon), + ("HubbleParam", cosmo.h), + ("BoxSize", boxsize), + ]), + ("%---- Output frequency", [ + ("OutputListOn", output_list_on), + ("TimeBetSnapshot", time_bet_snapshot), + ("TimeOfFirstSnapshot", time_of_first_snapshot), + ("TimeBetStatistics", time_bet_statistics), + ("NumFilesPerSnapshot", num_files_per_snapshot), + ("NumFilesWrittenInParallel", num_files_written_in_parallel), + ]), + ("%---- Accuracy of time integration", [ + ("TypeOfTimestepCriterion", type_of_timestep_criterion), + ("ErrTolIntAccuracy", err_tol_int_accuracy), + ("CourantFac", courant_fac), + ("MaxRMSDisplacementFac", max_rms_displacement_fac), + ("MaxSizeTimestep", max_size_timestep), + ("MinSizeTimestep", min_size_timestep), + ]), + ("%---- Gas temperatures", [ + ("InitGasTemp", init_gas_temp), + ("MinGasTemp", min_gas_temp), + ]), + ("%---- Tree algorithm, force accuracy, domain update frequency", [ + ("TypeOfOpeningCriterion", type_of_opening_criterion), + ("ErrTolTheta", err_tol_theta), + ("ErrTolForceAcc", err_tol_force_acc), + ("TreeDomainUpdateFrequency", tree_domain_update_frequency), + ]), + ("%---- Initial density estimate", [ + ("DesNumNgb", des_num_ngb), + ("MaxNumNgbDeviation", max_num_ngb_deviation), + ]), + ("%---- System of units", [ + ("UnitLength_in_cm", unit_length_in_cm), + ("UnitMass_in_g", unit_mass_in_g), + ("UnitVelocity_in_cm_per_s", unit_velocity_in_cm_per_s), + ("GravityConstantInternal", gravity_constant_internal), + ]), + ("%---- Gravitational softening lengths", [ + ("MinGasHsmlFractional", min_gas_hsml_fractional), + ]), + (None, [(f"Softening{t}", soft[t]) + for t in ("Gas", "Halo", "Disk", "Bulge", "Stars", "Bndry")]), + (None, [(f"Softening{t}MaxPhys", soft[t]) + for t in ("Gas", "Halo", "Disk", "Bulge", "Stars", "Bndry")]), + ("%---- non-common", [ + ("ArtBulkViscConst", art_bulk_visc_const), + ]), + ] + + if include_subfind: + sections.append((None, [ + ("ErrTolThetaSubfind", err_tol_theta_subfind), + ("DesLinkNgb", des_link_ngb), + ])) + + out = [] + for heading, pairs in sections: + if heading: + out.append(heading + "\n") + out.append(_params(pairs)) + out.append("\n") + return "".join(out).rstrip("\n") + "\n" + + +#: Compile-time options, as ``(token, enabled, comment)``. A disabled option is +#: emitted commented out, which is how Gadget's Config.sh expresses "off" while +#: keeping the option discoverable. +_CONFIG_SECTIONS = [ + ("Basic operation mode of code", [ + ("PERIODIC", True, None), + ("COOLING", False, None), + ("SFR", False, None), + ("SINKS", False, None), + ("UNEQUALSOFTENINGS", True, None), + ("NUM_THREADS=4", False, + "Now OpenMP works the same, so don't compile with OpenMP *and* PTHREADS !"), + ]), + ("Kernel Options", [ + ("QUINTIC_KERNEL", False, + "Implementation of the Morris 1996 quintic spline kernel, requires (3/2)^3 more neighbours !"), + ("TWODIMS", False, "Switch for 2D test problems"), + ("ONEDIM", False, "Switch for 1D test problems"), + ]), + ("Multi-Domain and Top-Level Tree options", [ + ("MULTIPLEDOMAINS=8", True, None), + ("TOPNODEFACTOR=3.0", False, None), + ("KD_HMAX_ESTIMATE", False, + "Alternative way to update HMAX within Tree nodes"), + ]), + ("Things that are always recommended", [ + ("PEANOHILBERT", True, None), + ("WALLCLOCK", True, None), + ("MYSORT", True, None), + ("AUTO_SWAP_ENDIAN_READIC", False, + "Enables automatic ENDIAN swapping for reading ICs"), + ("WRITE_KEY_FILES", False, "Enables writing key index files"), + ("WRITE_INFO_BLOCK", False, "Enables writing the INFO block"), + ("PERMUTATAION_OPTIMIZATION", False, None), + ("PROCESS_TIMES_OF_OUTPUTLIST", False, + "Chooses the outputtime closest to any global step"), + ("SYNCRONIZ_OUTPUT", False, "Writes output only at global time steps"), + ]), + ("Single/Double Precision", [ + ("DOUBLEPRECISION", True, None), + ("DOUBLEPRECISION_FFTW", True, None), + ("OUTPUT_IN_DOUBLEPRECISION", False, + "snapshot files will be written in double precision"), + ("INPUT_IN_DOUBLEPRECISION", False, None), + ]), + ("Invariance Test", [ + ("INVARIANCETEST", False, None), + ("INVARIANCETEST_SIZE1=2", False, None), + ("INVARIANCETEST_SIZE2=6", False, None), + ("FLTROUNDOFFREDUCTION", False, + "enables (expensive!) `double-double' round-off reduction in particle sums"), + ("SOFTDOUBLEDOUBLE", False, + "needs to be set if a C++ software implementation of 128bit double-double precision should be used"), + ]), +] + +#: SUBFIND-related options, emitted enabled or commented out together. +_SUBFIND_SECTION = [ + ("FOF", None), + ("FOF_PRIMARY_LINK_TYPES=2", "2^type for the primary dark matter type"), +] + +_SUBFIND_TRAILER = [ + ("FOF_GROUP_MIN_LEN=32", False, "default is 32"), + ("DENSITY_SPLIT_BY_TYPE=1+2+16+32", False, + "2^type for whch the densities should be calculated seperately"), + ("MAX_NGB_CHECK=3", False, + "Max numbers of neighbours for sattlepoint detection (default = 2)"), + ("SAVE_MASS_TAB", False, + "Saves the an additional array with the masses of the different components"), + ("SUBFINDSAVE_PARTICLELISTS", False, + "Saves also phase-space and type variables parallel to IDs"), + ("SO_VEL_DISPERSIONS", False, + "computes velocity dispersions for as part of FOF SO-properties"), + ("ORDER_SNAPSHOTS_BY_ID", False, None), + ("SAVE_HSML_IN_IC_ORDER", False, + "will store the hsml-values in the order of the particles in the IC file"), + ("ONLY_PRODUCE_HSML_FILES", False, "only carries out density estimate"), + ("KEEP_HSML_AS_GUESS", False, + "keep using hsml for gas particles in subfind_density"), + ("LINKLENGTH=0.16", False, "Linkinglength for FoF (default=0.2)"), + ("NO_GAS_CLOUDS", False, "Do not accept pure gaseous substructures"), + ("WRITE_SUB_IN_SNAP_FORMAT", False, "Save subfind results in snap format"), + ("LT_ADD_GAL_TO_SUB=12", False, + "Adds optical luminosities in 6 bands to subhalos"), + ("DUSTATT=11", False, + "Includes dust attenuation into the luminosity calculation (using 11 radial bins)"), + ("OBSERVER_FRAME", False, + "If defined, use CB07 Observer Frame Luminosities, otherwise CB07 Rest Frame Luminosities"), + ("SO_BAR_INFO", False, "Adds temperature, Lx, bfrac, etc to Groups"), + ("FSUBFINDCOUNT_BIG_HALOS=1e4", False, + "Adds extra blocks for Halos with M_TopHat > SUBFIND_COUNT_BIG_HALOS"), + ("KD_CHOOSE_PSUBFIND_LIMIT", False, + "Increases the limit for the parallel subfind to the maximum possible"), + ("KD_ALTERNATIVE_GROUP_SORT", False, + "Alternative way to sort the Groups/SubGroupe before writing"), + ("KD_CHOOSE_LINKING_LENGTH", False, "Special way to estimate the linking length"), + ("SUBFINDREAD_FOF", False, None), + ("SUBFINDCOLLECTIVE_STAGE1", False, None), + ("SUBFINDCOLLECTIVE_STAGE2", False, None), + ("SUBFINDALTERNATIVE_COLLECTIVE", False, None), + ("SUBFINDRESHUFFLE_CATALOGUE", False, None), + ("SUBFINDRESHUFFLE_CATALOGUE_WITH_VORONOI", False, None), + ("SUBFINDRESHUFFLE_AND_POTENTIAL", False, + "needs -DSUBFIND_RESHUFFLE_CATALOGUE and COMPUTE_POTENTIAL_ENERGY"), + ("SUBFINDDENSITY_AND_POTENTIAL", False, + "only calculated density and potential and write them into snapshot"), +] + + +#: Column trailing comments in Config.sh are aligned to. +_COMMENT_COLUMN = 35 + + +def _option(token, enabled, comment): + line = token if enabled else f"#{token}" + if comment: + # Always leave at least one space, so a token longer than the comment + # column does not end up with `#` jammed against it. + line += " " * max(1, _COMMENT_COLUMN - len(line)) + f"# {comment}" + return line + "\n" + + +def _heading(text): + return f"#--------------------------------------- {text}\n" + + +def config_sh( + *, + pmgrid=512, + include_subfind=False, + fof_secondary_link_types="4+8+16+32", + gridboost=2, + placehighresregion=2, + enlargeregion=1.2, +): + """Build a P-Gadget3 ``Config.sh``. + + Enabled options are emitted bare; disabled ones commented out, which is how + Gadget expresses "off" while keeping the option discoverable. + """ + out = [ + "#!/bin/bash\n", + "##################################################\n", + "# Enable/Disable compile-time options as needed #\n", + "##################################################\n", + ] + + def emit(section): + for token, enabled, comment in section: + out.append(_option(token, enabled, comment)) + + heading, options = _CONFIG_SECTIONS[0] + out.append(_heading(heading)) + emit(options) + + heading, options = _CONFIG_SECTIONS[1] + out.append(_heading(heading)) + emit(options) + + out.append(_heading("TreePM Options")) + emit([ + (f"PMGRID={int(pmgrid)}", True, None), + (f"GRIDBOOST={gridboost}", True, None), + ("#ASMTH=1.25", False, None), + ("RCUT=5.25", False, None), + (f"PLACEHIGHRESREGION={placehighresregion}", True, None), + (f"ENLARGEREGION={enlargeregion}", True, None), + ]) + + for heading, options in _CONFIG_SECTIONS[2:]: + out.append(_heading(heading)) + emit(options) + + out.append(_heading("On the fly FOF groupfinder")) + for token, comment in _SUBFIND_SECTION: + out.append(_option(token, include_subfind, comment)) + out.append(_option( + f"FOF_SECONDARY_LINK_TYPES={fof_secondary_link_types}", + include_subfind, + "2^type for the types linked to nearest primaries", + )) + out.append(_option("FOF_GROUP_MIN_LEN=32", False, "default is 32")) + out.append(_option("SUBFIND", include_subfind, None)) + for token, enabled, comment in _SUBFIND_TRAILER[1:]: + out.append(_option(token, enabled, comment)) + + return "".join(out) diff --git a/src/cme/writers/music.py b/src/cme/writers/music.py new file mode 100644 index 0000000..b58cd88 --- /dev/null +++ b/src/cme/writers/music.py @@ -0,0 +1,258 @@ +"""MUSIC configuration file generation. + +Extracted from the two module-level ``construct*conf`` functions in the ICs +tab, which each built the file with ~50 sequential ``f.write`` calls. + +Keys are left-justified to a fixed column before ``=``. MUSIC's parser is +whitespace-insensitive, so this is cosmetic, but it makes the output diffable. +""" + +#: Column width the ``key = value`` pairs are aligned to. +_KEY_WIDTH = 21 + +#: MUSIC's own names for the output formats the GUI offers. +_OUTPUT_FORMATS = {"music": "generic", "gadget": "gadget2"} + + +def output_format(name): + """Translate a GUI output format name to MUSIC's spelling. + + >>> output_format("gadget"), output_format("music"), output_format("enzo") + ('gadget2', 'generic', 'enzo') + """ + return _OUTPUT_FORMATS.get(str(name), str(name)) + + +def _section(title, pairs): + lines = [f"[{title}]"] + for key, value in pairs: + if value is None: + continue + lines.append(f"{key:<{_KEY_WIDTH}}= {value}") + return "\n".join(lines) + "\n" + + +def _bool(flag): + """MUSIC spells booleans ``yes`` and ``no``. + + Replaces `determineboolstr`, which returned an unbound local for anything + that was not exactly ``True`` or ``False`` -- a numpy bool, for instance. + + >>> _bool(True), _bool(False) + ('yes', 'no') + """ + return "yes" if flag else "no" + + +def refinement_seeds(seed_level, levelmax, halo_id, nrvir, base_seed): + """Per-level random seeds for a zoom refinement hierarchy. + + The coarse level takes ``base_seed``; each finer level takes a seed derived + deterministically from the halo ID and ``nrvir``, so a given halo always + resimulates with the same noise realisation. + + Returns a list of ``(level, seed)`` pairs. + + >>> refinement_seeds(9, 11, 190897, 4, 34567) + [(9, 34567), (10, 190937), (11, 191018)] + """ + seeds = [] + delta = int(nrvir) + for level in range(int(seed_level), int(levelmax) + 1): + # Incremented before use, and unused on the first iteration; preserved + # from the original so seeds stay reproducible against existing runs. + delta += delta * 2 + 1 + if level == int(seed_level): + seeds.append((level, int(base_seed))) + else: + seeds.append((level, int(halo_id) + int(delta))) + return seeds + + +def _cosmology_section(cosmo, transfer): + return _section( + "cosmology", + [ + ("Omega_m", cosmo.omega_m), + ("Omega_L", cosmo.omega_l), + ("Omega_b", cosmo.omega_b), + ("H0", cosmo.H0), + ("sigma_8", cosmo.sigma_8), + ("nspec", cosmo.n_spec), + ("transfer", transfer), + ], + ) + + +def _poisson_section( + fft_fine, accuracy, pre_smooth, post_smooth, smoother, laplace_order, grad_order +): + return _section( + "poisson", + [ + ("fft_fine", _bool(fft_fine)), + ("accuracy", accuracy), + ("pre_smooth", pre_smooth), + ("post_smooth", post_smooth), + ("smoother", smoother), + ("laplace_order", laplace_order), + ("grad_order", grad_order), + ], + ) + + +def _output_section(out_format, filename, num_files): + return _section( + "output", + [ + ("format", output_format(out_format)), + ("filename", filename), + ("gadget_num_files", num_files), + ], + ) + + +def resim_conf( + *, + cosmo, + boxlength, + zstart, + levelmin, + levelmin_TF, + levelmax, + padding, + overlap, + region, + region_point_file, + ref_center=None, + ref_extent=None, + align_top=False, + baryons=False, + use_2LPT=False, + use_2LLA=False, + periodic_TF=True, + transfer="eisenstein", + seed_level, + base_seed, + halo_id, + nrvir, + out_format="gadget", + ic_filename="./ics", + num_files=4, + fft_fine=True, + accuracy=1e-5, + pre_smooth=3, + post_smooth=3, + smoother="gs", + laplace_order=6, + grad_order=6, +): + """Build a MUSIC configuration for a zoom resimulation. + + ``cosmo`` is a :class:`cme.cosmology.Cosmology`. ``ref_center`` and + ``ref_extent`` are 3-tuples, written only for a box region -- an ellipsoid + is defined by the region point file alone. + """ + setup = [ + ("boxlength", boxlength), + ("zstart", zstart), + ("levelmin", levelmin), + ("levelmin_TF", levelmin_TF), + ("levelmax", levelmax), + ("padding", padding), + ("overlap", overlap), + ("region", region), + ] + if str(region) == "box": + setup += [ + ("ref_center", ",".join(str(v) for v in ref_center)), + ("ref_extent", ",".join(str(v) for v in ref_extent)), + ] + setup += [ + ("region_point_file", region_point_file), + ("align_top", _bool(align_top)), + ("baryons", _bool(baryons)), + ("use_2LPT", _bool(use_2LPT)), + ("use_2LLA", _bool(use_2LLA)), + ("periodic_TF", _bool(periodic_TF)), + ] + + seeds = refinement_seeds(seed_level, levelmax, halo_id, nrvir, base_seed) + + return "\n".join( + [ + _section("setup", setup), + _cosmology_section(cosmo, transfer), + _section("random", [(f"seed[{level}]", seed) for level, seed in seeds]), + _output_section(out_format, ic_filename, num_files), + _poisson_section( + fft_fine, accuracy, pre_smooth, post_smooth, + smoother, laplace_order, grad_order, + ), + ] + ) + + +def parent_conf( + *, + cosmo, + boxlength, + zstart, + level, + levelmin_TF, + padding, + overlap, + ref_center, + ref_extent, + seed, + align_top=False, + baryons=False, + use_2LPT=False, + use_2LLA=False, + periodic_TF=True, + transfer="eisenstein", + out_format="gadget", + ic_filename="./ics", + num_files=4, + fft_fine=True, + accuracy=1e-5, + pre_smooth=3, + post_smooth=3, + smoother="gs", + laplace_order=6, + grad_order=6, +): + """Build a MUSIC configuration for a uniform-resolution parent box. + + ``level`` sets both ``levelmin`` and ``levelmax``: a parent box is + unrefined, so there is a single seed at that level. + """ + setup = [ + ("boxlength", boxlength), + ("zstart", zstart), + ("levelmin", level), + ("levelmin_TF", levelmin_TF), + ("levelmax", level), + ("padding", padding), + ("overlap", overlap), + ("ref_center", ", ".join(str(v) for v in ref_center)), + ("ref_extent", ", ".join(str(v) for v in ref_extent)), + ("align_top", _bool(align_top)), + ("baryons", _bool(baryons)), + ("use_2LPT", _bool(use_2LPT)), + ("use_2LLA", _bool(use_2LLA)), + ("periodic_TF", _bool(periodic_TF)), + ] + + return "\n".join( + [ + _section("setup", setup), + _cosmology_section(cosmo, transfer), + _section("random", [(f"seed[{int(level)}]", int(seed))]), + _output_section(out_format, ic_filename, num_files), + _poisson_section( + fft_fine, accuracy, pre_smooth, post_smooth, + smoother, laplace_order, grad_order, + ), + ] + ) diff --git a/src/cme/writers/submit.py b/src/cme/writers/submit.py new file mode 100644 index 0000000..eb7415f --- /dev/null +++ b/src/cme/writers/submit.py @@ -0,0 +1,142 @@ +"""Batch scheduler submission scripts for PBS/Torque and SLURM. + +Extracted from ``_subscript_button_fired`` on the Gadget tab, which built both +script flavours inline and carried three defects that this module's typing +makes impossible: + +* the SLURM branch concatenated integer traits with strings + (``'#SBATCH -n ' + self.SLURMcores``), raising ``TypeError`` -- so SLURM + submission had never worked; +* the SLURM ``mpirun`` line was built from the *PBS* core count, so a SLURM job + would have launched with the wrong number of ranks; +* enabling SUBFIND switched the PBS command to the ``_sub`` binary and + ``param_sub.txt`` but left the SLURM command pointing at the plain binary, + because both branches of ``_SLURMcores_changed`` were identical. + +Rank counts are formatted with ``int()`` here, and one shared +:func:`mpirun_command` serves both schedulers. +""" + +#: Gadget's restart flag for "run FOF/SUBFIND on an existing snapshot". +SUBFIND_RESTART_FLAG = 3 + + +def mpirun_command( + *, + ranks, + include_subfind=False, + binary="P-Gadget3", + param="param.txt", + out="OUTPUT", + err="ERROR", + subfind_snapshot=63, +): + """The ``mpirun`` line that launches Gadget. + + With ``include_subfind`` the ``_sub`` binary and ``param_sub.txt`` are used, + the restart flag and snapshot number are appended, and the log filenames + gain a ``sub`` suffix -- matching the existing convention. + + >>> mpirun_command(ranks=64) + 'mpirun -np 64 ./P-Gadget3 ./param.txt 1>OUTPUT 2>ERROR' + >>> mpirun_command(ranks=64, include_subfind=True) + 'mpirun -np 64 ./P-Gadget3_sub ./param_sub.txt 3 63 1>OUTPUTsub 2>ERRORsub' + """ + if include_subfind: + stem, ext = param.rsplit(".", 1) if "." in param else (param, None) + param = f"{stem}_sub" + (f".{ext}" if ext else "") + binary = f"{binary}_sub" + args = f" {SUBFIND_RESTART_FLAG} {int(subfind_snapshot)}" + out, err = f"{out}sub", f"{err}sub" + else: + args = "" + + return ( + f"mpirun -np {int(ranks)} ./{binary} ./{param}{args} 1>{out} 2>{err}" + ) + + +def slurm_script( + *, + job_name, + work_dir, + execute, + cores, + minutes, + memory_mb, + queue, + email=None, +): + """A SLURM batch script. + + ``memory_mb`` is per CPU, matching ``--mem-per-cpu``. + """ + lines = [ + "#!/bin/bash", + f"#SBATCH -n {int(cores)}", + f"#SBATCH -o {job_name}.o%j", + f"#SBATCH -e {job_name}.e%j", + f"#SBATCH -t {int(minutes)}", + f"#SBATCH -p {queue}", + f"#SBATCH --mem-per-cpu={int(memory_mb)}", + ] + if email: + lines += [ + f"#SBATCH --mail-user={email}", + "#SBATCH --mail-type=begin", + "#SBATCH --mail-type=end", + ] + lines += [ + f"#SBATCH -J {job_name}", + "", + f"cd {work_dir}", + "", + execute, + "", + ] + return "\n".join(lines) + + +def pbs_script( + *, + job_name, + work_dir, + execute, + cores_per_node, + nodes, + queue, + email=None, + extra_flags="", + extra_lines="", +): + """A PBS/Torque batch script.""" + lines = [ + "#!/bin/csh", + "#PBS -k eo", + f"#PBS -l nodes={int(nodes)}:ppn={int(cores_per_node)}", + f"#PBS -N {job_name}", + ] + if email: + lines.append(f"#PBS -M {email}") + lines += [ + f"#PBS -o {work_dir}/{job_name}.out", + f"#PBS -e {work_dir}/{job_name}.err", + f"#PBS -q {queue}", + ] + if extra_flags: + lines.append(extra_flags) + lines.append("") + if extra_lines: + lines += [extra_lines, ""] + lines += [f"cd {work_dir}", "", execute, ""] + return "\n".join(lines) + + +def script(scheduler, **kwargs): + """Dispatch to :func:`slurm_script` or :func:`pbs_script` by name.""" + scheduler = str(scheduler).upper() + if scheduler == "SLURM": + return slurm_script(**kwargs) + if scheduler == "PBS": + return pbs_script(**kwargs) + raise ValueError(f"unknown scheduler {scheduler!r}; expected 'SLURM' or 'PBS'") diff --git a/tests/test_writers_gadget.py b/tests/test_writers_gadget.py new file mode 100644 index 0000000..0e92f4b --- /dev/null +++ b/tests/test_writers_gadget.py @@ -0,0 +1,204 @@ +import pytest + +from cme import cosmology +from cme.writers import gadget + + +def parse(text): + """Gadget's parser is whitespace-separated `key value`, `%` for comments.""" + params = {} + for line in text.splitlines(): + line = line.strip() + if not line or line.startswith("%"): + continue + key, value = line.split(None, 1) + assert key not in params, f"duplicate parameter {key!r}" + params[key] = value + return params + + +PARAM = dict(cosmo=cosmology.PLANCK, boxsize=100.0, levelmax=11) + + +class TestSoftenings: + def test_halo_is_one_fortieth_of_the_grid_spacing(self): + soft = gadget.softenings(100.0, 11) + assert soft["Halo"] == pytest.approx(100.0 / 2**11 / 40) + + def test_each_type_doubles_the_previous(self): + soft = gadget.softenings(100.0, 11) + assert soft["Disk"] == pytest.approx(2 * soft["Halo"]) + assert soft["Bulge"] == pytest.approx(2 * soft["Disk"]) + assert soft["Stars"] == pytest.approx(2 * soft["Bulge"]) + assert soft["Bndry"] == pytest.approx(2 * soft["Stars"]) + + def test_gas_is_zero_for_dark_matter_only_runs(self): + assert gadget.softenings(100.0, 11)["Gas"] == 0.0 + + def test_finer_levelmax_gives_smaller_softening(self): + assert ( + gadget.softenings(100.0, 12)["Halo"] + < gadget.softenings(100.0, 11)["Halo"] + ) + + def test_reproduces_the_hardcoded_defaults(self): + # The Gadget tab shipped these as literal defaults for a 100 Mpc/h box + # at levelmax 11; they are the reference for the scaling above. + soft = gadget.softenings(100.0, 11) + assert soft["Halo"] == pytest.approx(0.001220703, rel=1e-6) + assert soft["Disk"] == pytest.approx(0.002441406, rel=1e-6) + + +class TestPmgrid: + @pytest.mark.parametrize( + "levelmax,expected", + [(7, 256), (10, 256), (11, 256), (12, 512), (13, 512), (14, 1024), (15, 1024)], + ) + def test_thresholds(self, levelmax, expected): + assert gadget.pmgrid_for_levelmax(levelmax) == expected + + def test_accepts_strings(self): + # The Gadget tab's lmax is an Enum of strings. + assert gadget.pmgrid_for_levelmax("12") == 512 + + +class TestParamFile: + def test_has_no_duplicate_keys(self): + parse(gadget.param_file(**PARAM)) # asserts internally + + def test_hubble_param_is_dimensionless_h(self): + # The GUI stored H0 in a trait named HubbleParam and divided by 100 on + # the way out, so the displayed and written values disagreed. Gadget + # wants h. + params = parse(gadget.param_file(**PARAM)) + assert float(params["HubbleParam"]) == pytest.approx(0.6711) + + def test_cosmological_parameters_come_from_the_cosmology(self): + params = parse(gadget.param_file(**PARAM)) + assert float(params["Omega0"]) == pytest.approx(0.3175) + assert float(params["OmegaLambda"]) == pytest.approx(0.6825) + + def test_omega_baryon_is_independent_of_the_cosmology(self): + # Dark-matter-only runs zero this while keeping the rest of the set. + params = parse(gadget.param_file(**PARAM)) + assert float(params["OmegaBaryon"]) == 0.0 + params = parse(gadget.param_file(**PARAM, omega_baryon=0.049)) + assert float(params["OmegaBaryon"]) == pytest.approx(0.049) + + def test_softenings_are_derived_from_boxsize_and_levelmax(self): + params = parse(gadget.param_file(**PARAM)) + expected = gadget.softenings(100.0, 11) + assert float(params["SofteningHalo"]) == pytest.approx(expected["Halo"]) + assert float(params["SofteningHaloMaxPhys"]) == pytest.approx(expected["Halo"]) + + def test_subfind_parameters_only_present_when_requested(self): + without = parse(gadget.param_file(**PARAM)) + assert "ErrTolThetaSubfind" not in without + assert "DesLinkNgb" not in without + + with_sub = parse(gadget.param_file(**PARAM, include_subfind=True)) + assert float(with_sub["ErrTolThetaSubfind"]) == pytest.approx(0.7) + assert int(with_sub["DesLinkNgb"]) == 20 + + def test_contains_every_parameter_the_gui_exposed(self): + params = parse(gadget.param_file(**PARAM, include_subfind=True)) + required = { + "InitCondFile", "OutputDir", "EnergyFile", "InfoFile", "TimingsFile", + "CpuFile", "RestartFile", "SnapshotFileBase", "OutputListFilename", + "TimebinFile", "ICFormat", "SnapFormat", "TimeLimitCPU", + "CpuTimeBetRestartFile", "ResubmitOn", "ResubmitCommand", "MaxMemSize", + "PartAllocFactor", "BufferSize", "TimeBegin", "TimeMax", + "ComovingIntegrationOn", "PeriodicBoundariesOn", "CoolingOn", + "StarformationOn", "Omega0", "OmegaLambda", "OmegaBaryon", + "HubbleParam", "BoxSize", "OutputListOn", "TimeBetSnapshot", + "TimeOfFirstSnapshot", "TimeBetStatistics", "NumFilesPerSnapshot", + "NumFilesWrittenInParallel", "TypeOfTimestepCriterion", + "ErrTolIntAccuracy", "CourantFac", "MaxRMSDisplacementFac", + "MaxSizeTimestep", "MinSizeTimestep", "InitGasTemp", "MinGasTemp", + "TypeOfOpeningCriterion", "ErrTolTheta", "ErrTolForceAcc", + "TreeDomainUpdateFrequency", "DesNumNgb", "MaxNumNgbDeviation", + "UnitLength_in_cm", "UnitMass_in_g", "UnitVelocity_in_cm_per_s", + "GravityConstantInternal", "MinGasHsmlFractional", "ArtBulkViscConst", + "ErrTolThetaSubfind", "DesLinkNgb", + } + for name in ("Gas", "Halo", "Disk", "Bulge", "Stars", "Bndry"): + required |= {f"Softening{name}", f"Softening{name}MaxPhys"} + assert required - set(params) == set() + + def test_explicit_softening_overrides_the_derived_one(self): + custom = dict.fromkeys( + ("Gas", "Halo", "Disk", "Bulge", "Stars", "Bndry"), 1.5 + ) + params = parse(gadget.param_file(**PARAM, softening=custom)) + assert float(params["SofteningHalo"]) == 1.5 + + +class TestExpansionList: + def test_one_line_per_output(self): + text = gadget.expansion_list([(0.02, 1), (0.5, 0), (1.0, 1)]) + assert text.splitlines() == ["0.02 1", "0.5 0", "1.0 1"] + + def test_flag_is_an_integer(self): + assert gadget.expansion_list([(0.5, True)]).strip() == "0.5 1" + + def test_empty_list_gives_empty_file(self): + assert gadget.expansion_list([]) == "" + + +class TestConfigSh: + def enabled(self, text): + """The set of active compile-time tokens, with trailing comments stripped.""" + tokens = set() + for line in text.splitlines(): + line = line.strip() + if not line or line.startswith("#"): + continue + tokens.add(line.split("#", 1)[0].strip()) + return tokens + + def test_every_commented_option_keeps_a_space_before_its_comment(self): + # `FOO=1# comment` is ambiguous to naive parsers of Config.sh. + for line in gadget.config_sh(include_subfind=True).splitlines(): + body, _, comment = line.partition("#") + if body.strip() and comment: + assert body.endswith(" "), line + + def test_starts_with_a_shebang(self): + assert gadget.config_sh().startswith("#!/bin/bash\n") + + def test_pmgrid_is_emitted_enabled(self): + assert "PMGRID=512" in self.enabled(gadget.config_sh(pmgrid=512)) + assert "PMGRID=1024" in self.enabled(gadget.config_sh(pmgrid=1024)) + + def test_options_always_on(self): + on = self.enabled(gadget.config_sh()) + for token in ("PERIODIC", "UNEQUALSOFTENINGS", "PEANOHILBERT", + "WALLCLOCK", "MYSORT", "DOUBLEPRECISION", + "DOUBLEPRECISION_FFTW", "PLACEHIGHRESREGION=2"): + assert token in on + + def test_options_always_off(self): + on = self.enabled(gadget.config_sh()) + for token in ("COOLING", "SFR", "SINKS", "TWODIMS", "ONEDIM"): + assert token not in on + + def test_subfind_disabled_by_default(self): + on = self.enabled(gadget.config_sh()) + assert "FOF" not in on + assert "SUBFIND" not in on + + def test_subfind_enables_fof_and_subfind_together(self): + on = self.enabled(gadget.config_sh(include_subfind=True)) + assert "FOF" in on + assert "SUBFIND" in on + assert "FOF_PRIMARY_LINK_TYPES=2" in on + + def test_secondary_link_types_are_configurable(self): + text = gadget.config_sh(include_subfind=True, + fof_secondary_link_types="4+8") + assert "FOF_SECONDARY_LINK_TYPES=4+8" in self.enabled(text) + + def test_disabled_options_stay_discoverable_as_comments(self): + # Gadget's Config.sh convention: "off" means present but commented. + assert "#COOLING" in gadget.config_sh() + assert "#SUBFIND" in gadget.config_sh() diff --git a/tests/test_writers_music.py b/tests/test_writers_music.py new file mode 100644 index 0000000..b64cae0 --- /dev/null +++ b/tests/test_writers_music.py @@ -0,0 +1,205 @@ +import configparser + +import pytest + +from cme import cosmology +from cme.writers import music + + +def parse(text): + """MUSIC configs are INI, so the emitted text must survive a real parser.""" + parser = configparser.ConfigParser() + parser.read_string(text) + return parser + + +RESIM = dict( + cosmo=cosmology.PLANCK, + boxlength=100, + zstart=127, + levelmin=7, + levelmin_TF=9, + levelmax=11, + padding=7, + overlap=4, + region="ellipsoid", + region_point_file="/data/ics/lagr/H190897NRVIR4", + seed_level=9, + base_seed=34567, + halo_id=190897, + nrvir=4, +) + + +class TestOutputFormat: + def test_translates_gui_names_to_music_names(self): + assert music.output_format("gadget") == "gadget2" + assert music.output_format("music") == "generic" + + def test_passes_other_formats_through(self): + for name in ("enzo", "grafic2", "generic", "tipsy"): + assert music.output_format(name) == name + + +class TestRefinementSeeds: + def test_coarse_level_uses_the_base_seed(self): + seeds = dict(music.refinement_seeds(9, 11, 190897, 4, 34567)) + assert seeds[9] == 34567 + + def test_one_seed_per_level_inclusive_of_levelmax(self): + seeds = music.refinement_seeds(9, 12, 190897, 4, 34567) + assert [level for level, _ in seeds] == [9, 10, 11, 12] + + def test_is_deterministic_for_a_given_halo(self): + assert music.refinement_seeds(9, 11, 190897, 4, 34567) == ( + music.refinement_seeds(9, 11, 190897, 4, 34567) + ) + + def test_differs_between_halos_and_between_nrvir(self): + base = music.refinement_seeds(9, 11, 190897, 4, 34567) + assert base != music.refinement_seeds(9, 11, 190898, 4, 34567) + assert base != music.refinement_seeds(9, 11, 190897, 5, 34567) + + def test_reproduces_the_original_derivation(self): + # Pinned against the original inline arithmetic: delta starts at nrvir + # and is updated as delta += delta*2 + 1 before each level is emitted. + assert music.refinement_seeds(9, 11, 190897, 4, 34567) == [ + (9, 34567), (10, 190937), (11, 191018), + ] + + +class TestResimConf: + def test_is_valid_ini_with_the_expected_sections(self): + parser = parse(music.resim_conf(**RESIM)) + assert set(parser.sections()) == { + "setup", "cosmology", "random", "output", "poisson" + } + + def test_setup_values(self): + setup = parse(music.resim_conf(**RESIM))["setup"] + assert setup["boxlength"] == "100" + assert setup["zstart"] == "127" + assert setup["levelmin"] == "7" + assert setup["levelmax"] == "11" + assert setup["padding"] == "7" + assert setup["overlap"] == "4" + assert setup["region"] == "ellipsoid" + assert setup["region_point_file"] == "/data/ics/lagr/H190897NRVIR4" + + def test_booleans_are_spelled_yes_and_no(self): + setup = parse(music.resim_conf(**RESIM, baryons=True, use_2LPT=False))["setup"] + assert setup["baryons"] == "yes" + assert setup["use_2lpt"] == "no" + assert setup["periodic_tf"] == "yes" + + def test_ellipsoid_region_omits_ref_center_and_extent(self): + # An ellipsoid is defined by the point file alone. + setup = parse(music.resim_conf(**RESIM))["setup"] + assert "ref_center" not in setup + assert "ref_extent" not in setup + + def test_box_region_includes_ref_center_and_extent(self): + kwargs = dict(RESIM, region="box", + ref_center=(0.51, 0.52, 0.53), + ref_extent=(0.21, 0.22, 0.23)) + setup = parse(music.resim_conf(**kwargs))["setup"] + assert setup["ref_center"] == "0.51,0.52,0.53" + assert setup["ref_extent"] == "0.21,0.22,0.23" + + def test_cosmology_section_uses_H0_not_h(self): + # MUSIC wants H0 in km/s/Mpc. + cosmo = parse(music.resim_conf(**RESIM))["cosmology"] + assert float(cosmo["h0"]) == pytest.approx(67.11) + assert float(cosmo["omega_m"]) == pytest.approx(0.3175) + assert float(cosmo["sigma_8"]) == pytest.approx(0.8344) + + def test_random_section_has_one_seed_per_refinement_level(self): + random = parse(music.resim_conf(**RESIM))["random"] + assert sorted(random) == ["seed[10]", "seed[11]", "seed[9]"] + + def test_output_format_is_translated(self): + output = parse(music.resim_conf(**RESIM, out_format="gadget"))["output"] + assert output["format"] == "gadget2" + + def test_golden(self): + assert music.resim_conf(**RESIM) == GOLDEN_RESIM + + +class TestParentConf: + PARENT = dict( + cosmo=cosmology.PLANCK, + boxlength=100, + zstart=127, + level=9, + levelmin_TF=9, + padding=8, + overlap=4, + ref_center=(0.5, 0.5, 0.5), + ref_extent=(0.2, 0.2, 0.2), + seed=34567, + ) + + def test_is_unrefined_so_levelmin_equals_levelmax(self): + setup = parse(music.parent_conf(**self.PARENT))["setup"] + assert setup["levelmin"] == setup["levelmax"] == "9" + + def test_has_exactly_one_seed(self): + random = parse(music.parent_conf(**self.PARENT))["random"] + assert list(random) == ["seed[9]"] + assert random["seed[9]"] == "34567" + + def test_always_writes_ref_center_and_extent(self): + setup = parse(music.parent_conf(**self.PARENT))["setup"] + assert setup["ref_center"] == "0.5, 0.5, 0.5" + + def test_has_no_region_point_file(self): + assert "region_point_file" not in parse( + music.parent_conf(**self.PARENT) + )["setup"] + + +GOLDEN_RESIM = """\ +[setup] +boxlength = 100 +zstart = 127 +levelmin = 7 +levelmin_TF = 9 +levelmax = 11 +padding = 7 +overlap = 4 +region = ellipsoid +region_point_file = /data/ics/lagr/H190897NRVIR4 +align_top = no +baryons = no +use_2LPT = no +use_2LLA = no +periodic_TF = yes + +[cosmology] +Omega_m = 0.3175 +Omega_L = 0.6825 +Omega_b = 0.049 +H0 = 67.11 +sigma_8 = 0.8344 +nspec = 0.9624 +transfer = eisenstein + +[random] +seed[9] = 34567 +seed[10] = 190937 +seed[11] = 191018 + +[output] +format = gadget2 +filename = ./ics +gadget_num_files = 4 + +[poisson] +fft_fine = yes +accuracy = 1e-05 +pre_smooth = 3 +post_smooth = 3 +smoother = gs +laplace_order = 6 +grad_order = 6 +""" diff --git a/tests/test_writers_submit.py b/tests/test_writers_submit.py new file mode 100644 index 0000000..8dd8324 --- /dev/null +++ b/tests/test_writers_submit.py @@ -0,0 +1,153 @@ +import pytest + +from cme.writers import submit + + +class TestMpirunCommand: + def test_plain_run(self): + assert submit.mpirun_command(ranks=64) == ( + "mpirun -np 64 ./P-Gadget3 ./param.txt 1>OUTPUT 2>ERROR" + ) + + def test_subfind_run_uses_the_sub_binary_param_and_logs(self): + assert submit.mpirun_command(ranks=64, include_subfind=True) == ( + "mpirun -np 64 ./P-Gadget3_sub ./param_sub.txt 3 63 " + "1>OUTPUTsub 2>ERRORsub" + ) + + def test_subfind_snapshot_is_configurable(self): + cmd = submit.mpirun_command( + ranks=8, include_subfind=True, subfind_snapshot=127 + ) + assert " 3 127 " in cmd + + def test_ranks_accept_strings_without_concatenating(self): + # AUDIT 2.1.1: the SLURM branch concatenated an int trait with a string. + # Coercing here means neither a str nor an int input can break it. + assert "-np 512" in submit.mpirun_command(ranks="512") + assert "-np 512" in submit.mpirun_command(ranks=512) + + def test_log_filenames_are_configurable(self): + cmd = submit.mpirun_command(ranks=8, out="LOG", err="ERR") + assert "1>LOG 2>ERR" in cmd + + +class TestSlurmScript: + KWARGS = dict( + job_name="H19EP7L11N4", + work_dir="/data/halos/H190897/H190897_BE_Z127_P7_LN7_LX11_O4_NV4", + execute="mpirun -np 64 ./P-Gadget3 ./param.txt 1>OUTPUT 2>ERROR", + cores=64, + minutes=5000, + memory_mb=3500, + queue="itc_cluster", + email="someone@example.org", + ) + + def test_integer_fields_do_not_raise(self): + # AUDIT 2.1.1: this is the regression test for the TypeError that made + # SLURM submission impossible -- `'#SBATCH -n ' + self.SLURMcores`. + text = submit.slurm_script(**self.KWARGS) + assert "#SBATCH -n 64" in text + assert "#SBATCH -t 5000" in text + assert "#SBATCH --mem-per-cpu=3500" in text + + def test_accepts_string_valued_traits_too(self): + text = submit.slurm_script( + **dict(self.KWARGS, cores="64", minutes="5000", memory_mb="3500") + ) + assert "#SBATCH -n 64" in text + + def test_starts_with_bash_shebang(self): + assert submit.slurm_script(**self.KWARGS).startswith("#!/bin/bash\n") + + def test_changes_into_the_run_directory_before_executing(self): + lines = [ + ln for ln in submit.slurm_script(**self.KWARGS).splitlines() if ln.strip() + ] + assert lines.index(f"cd {self.KWARGS['work_dir']}") < lines.index( + self.KWARGS["execute"] + ) + + def test_mail_directives_omitted_without_an_email(self): + text = submit.slurm_script(**dict(self.KWARGS, email=None)) + assert "--mail" not in text + + def test_queue_and_job_name(self): + text = submit.slurm_script(**self.KWARGS) + assert "#SBATCH -p itc_cluster" in text + assert "#SBATCH -J H19EP7L11N4" in text + + +class TestPbsScript: + KWARGS = dict( + job_name="H19EP7L11N4", + work_dir="/data/halos/H190897/run", + execute="mpirun -np 24 ./P-Gadget3 ./param.txt 1>OUTPUT 2>ERROR", + cores_per_node=8, + nodes=3, + queue="default", + email="someone@example.org", + ) + + def test_resource_line_multiplies_out_as_nodes_and_ppn(self): + assert "#PBS -l nodes=3:ppn=8" in submit.pbs_script(**self.KWARGS) + + def test_uses_csh(self): + assert submit.pbs_script(**self.KWARGS).startswith("#!/bin/csh\n") + + def test_log_paths_are_absolute(self): + text = submit.pbs_script(**self.KWARGS) + assert "#PBS -o /data/halos/H190897/run/H19EP7L11N4.out" in text + assert "#PBS -e /data/halos/H190897/run/H19EP7L11N4.err" in text + + def test_extra_flags_and_lines_are_included(self): + text = submit.pbs_script( + **dict(self.KWARGS, extra_flags="#PBS -m ae", + extra_lines="source /opt/torque/etc/openmpi-setup.sh") + ) + assert "#PBS -m ae" in text + assert "source /opt/torque/etc/openmpi-setup.sh" in text + + def test_extra_fields_omitted_when_blank(self): + text = submit.pbs_script(**self.KWARGS) + assert "\n\n\n" not in text + + +class TestDispatch: + def test_selects_by_name_case_insensitively(self): + common = dict(job_name="j", work_dir="/w", execute="run") + slurm = submit.script( + "slurm", **common, cores=8, minutes=10, memory_mb=100, queue="q" + ) + assert slurm.startswith("#!/bin/bash") + pbs = submit.script( + "PBS", **common, cores_per_node=8, nodes=1, queue="q" + ) + assert pbs.startswith("#!/bin/csh") + + def test_rejects_unknown_scheduler(self): + with pytest.raises(ValueError, match="unknown scheduler"): + submit.script("lsf", job_name="j", work_dir="/w", execute="run") + + +class TestSubfindConsistencyAcrossSchedulers: + """AUDIT 2.2.5 / 2.2.6. + + Both schedulers must derive their rank count from their own core setting, + and both must switch to the `_sub` binary when SUBFIND is enabled. The GUI + built the SLURM command from the PBS core count, and left the SLURM command + on the plain binary because both branches of `_SLURMcores_changed` were + identical. + """ + + def test_each_scheduler_uses_its_own_rank_count(self): + slurm_ranks, pbs_ranks = 512, 24 + assert f"-np {slurm_ranks}" in submit.mpirun_command(ranks=slurm_ranks) + assert f"-np {pbs_ranks}" in submit.mpirun_command(ranks=pbs_ranks) + + @pytest.mark.parametrize("ranks", [24, 512]) + def test_subfind_switches_binary_regardless_of_rank_count(self, ranks): + cmd = submit.mpirun_command(ranks=ranks, include_subfind=True) + assert "P-Gadget3_sub" in cmd + assert "param_sub.txt" in cmd From b97ab6a2c94f8e0c5c6dd126741163da3d4cb10e Mon Sep 17 00:00:00 2001 From: Brendan Griffen Date: Mon, 27 Jul 2026 22:19:28 +1000 Subject: [PATCH 10/19] refactor: extract candidate selection into cme.science.candidates The isolation-criteria scan -- the scientific heart of the tool -- lived inside a button handler, interleaved with figure clearing and axis labelling, and could not be exercised without a display and a real parent catalogue. It is now a pure function over a host catalogue with 30 tests against synthetic inputs with known answers. Fixes: - AUDIT 2.2.3: the results array was seeded with np.zeros((1, 6)) and grown by vstack, so every selection carried a phantom zero-mass candidate at the origin. It was plotted, included in the array view, and -- because the "add halo" guard tested `len(...) != 1` to detect the sentinel -- could be added to the working sample as halo ID 0. select() now returns an empty array when nothing qualifies, and the guard tests for no rows. - AUDIT 2.1.8: _clear_button_fired assigned to `self.main.mergertreetab.initstab`, a stray attribute on the wrong object, so clearing the sample left the ICs tab holding the old one. It also never cleared the contamination or Gadget tabs. Both paths now go through one _publish_sample(), which was also duplicated (the merger tree tab was assigned twice in _addhalo_button_fired and the contamination tab omitted). - AUDIT 4.5: the six position and mass arrays were re-extracted from the catalogue inside the per-candidate loop, making the scan O(N^2) in the base sample, and each accepted candidate triggered a full-array vstack copy. Extraction now happens once and results accumulate in a list. A 4000-halo sample completes in under a second. The exclusion zones are now first-class: an ExclusionZone is either an absolute mass threshold or a multiple of the candidate's own mass, and the relative form excludes the candidate from its own comparison (a halo is zero distance from itself). The GUI supported both per zone but the code only ever applied a relative threshold to zone 1 and absolute ones to zones 2 and 3. Tests cover the hubble conversion on both mass and distance, three-dimensional separation, per-candidate threshold scaling, the default Caterpillar criteria as an integration case, and the candidates.dat round trip. --- src/cme/science/__init__.py | 7 + src/cme/science/candidates.py | 161 +++++++++++++++++++ src/cme/ui/candidates.py | 218 +++++++++++--------------- tests/test_science_candidates.py | 257 +++++++++++++++++++++++++++++++ 4 files changed, 514 insertions(+), 129 deletions(-) create mode 100644 src/cme/science/__init__.py create mode 100644 src/cme/science/candidates.py create mode 100644 tests/test_science_candidates.py diff --git a/src/cme/science/__init__.py b/src/cme/science/__init__.py new file mode 100644 index 0000000..2761906 --- /dev/null +++ b/src/cme/science/__init__.py @@ -0,0 +1,7 @@ +"""Display-free computation. + +Functions here take arrays and catalogues and return arrays and catalogues. +They do not touch matplotlib, Traits, or the filesystem, so they can be tested +against synthetic inputs with a known answer and reused to drive a suite +non-interactively. +""" diff --git a/src/cme/science/candidates.py b/src/cme/science/candidates.py new file mode 100644 index 0000000..a3ba289 --- /dev/null +++ b/src/cme/science/candidates.py @@ -0,0 +1,161 @@ +"""Selection of isolated Milky-Way-analogue host haloes. + +A candidate must fall inside a virial mass range and satisfy a set of isolation +criteria, each of the form "no halo above mass M lies within distance R". + +Extracted from the Candidates tab, where the same logic was interleaved with +matplotlib calls inside a button handler. Three defects came with it: + +* the results array was seeded with ``np.zeros((1, 6))`` and grown by + ``vstack``, so every selection carried a phantom zero-mass, zero-position + candidate that was plotted, counted in the tally, and accepted by the "add + halo" membership test (AUDIT 2.2.3); +* the six position and mass arrays were re-extracted from the catalogue inside + the per-candidate loop, making the scan O(N^2) in the base sample + (AUDIT 4.5); +* ``vstack`` per accepted candidate copied the whole result set each time. + +Masses are expected in Msol/h and positions in Mpc/h, as Rockstar writes them; +``hubble`` converts both to physical units before any comparison. +""" + +from dataclasses import dataclass + +import numpy as np + +#: Columns returned by :func:`select`, in order. +COLUMNS = ("id", "mvir", "rvir", "posX", "posY", "posZ") + + +@dataclass(frozen=True) +class ExclusionZone: + """One isolation criterion: no halo above a mass within ``distance``. + + Exactly one of ``mass`` (an absolute threshold in Msol) or ``mass_multiple`` + (a multiple of the candidate's own mass) must be given. A relative + threshold excludes the candidate itself from the comparison, since a halo is + trivially within zero distance of itself. + """ + + distance: float + mass: float = None + mass_multiple: float = None + + def __post_init__(self): + if (self.mass is None) == (self.mass_multiple is None): + raise ValueError( + "specify exactly one of mass or mass_multiple, " + f"got mass={self.mass!r}, mass_multiple={self.mass_multiple!r}" + ) + + @property + def is_relative(self): + return self.mass_multiple is not None + + +def _physical(catalogue, hubble): + """Extract positions and masses once, in physical units.""" + return ( + np.asarray(catalogue["id"]), + np.asarray(catalogue["mvir"], dtype=np.float64) / hubble, + np.asarray(catalogue["rvir"], dtype=np.float64), + np.asarray(catalogue["posX"], dtype=np.float64), + np.asarray(catalogue["posY"], dtype=np.float64), + np.asarray(catalogue["posZ"], dtype=np.float64), + ) + + +def _min_distance(x, y, z, xs, ys, zs, hubble, exclude=None): + """Distance in Mpc from one point to the nearest of many, or inf if none.""" + if exclude is not None: + xs, ys, zs = xs[exclude], ys[exclude], zs[exclude] + if xs.size == 0: + return np.inf + return float( + np.sqrt((x - xs) ** 2 + (y - ys) ** 2 + (z - zs) ** 2).min() / hubble + ) + + +def select(hosts, mass_range, zones, hubble): + """Return the isolated candidates in ``hosts`` as a structured array. + + ``hosts`` is a host-halo catalogue: any mapping of the columns in + :data:`COLUMNS` to sequences, such as the DataFrame from + ``RSDataReader.get_hosts()``. + + ``mass_range`` is ``(low, high)`` in Msol, applied to the physical virial + mass. ``zones`` is an iterable of :class:`ExclusionZone`. + + The returned array has the dtype implied by :data:`COLUMNS` and is empty -- + not one row of zeros -- when nothing qualifies. + + >>> import numpy as np + >>> cat = {"id": [1], "mvir": [1e12], "rvir": [200.0], + ... "posX": [10.0], "posY": [10.0], "posZ": [10.0]} + >>> out = select(cat, (7e11, 7e12), [], hubble=1.0) + >>> out["id"], len(out) + (array([1]), 1) + """ + low, high = mass_range + zones = list(zones) + + ids, mvir, rvir, x, y, z = _physical(hosts, hubble) + + in_range = (mvir > low) & (mvir < high) + candidate_index = np.flatnonzero(in_range) + + # Pre-compute the comparison population for every absolute-threshold zone + # once, rather than per candidate. + absolute = {} + for zone in zones: + if not zone.is_relative: + absolute[id(zone)] = mvir > zone.mass + + accepted = [] + for i in candidate_index: + for zone in zones: + if zone.is_relative: + above = mvir >= zone.mass_multiple * mvir[i] + # A halo is zero distance from itself, so drop it. + above &= ids != ids[i] + else: + above = absolute[id(zone)] + + if _min_distance(x[i], y[i], z[i], x, y, z, hubble, above) < zone.distance: + break + else: + accepted.append((ids[i], mvir[i], rvir[i], x[i], y[i], z[i])) + + dtype = [("id", np.int64)] + [(name, np.float64) for name in COLUMNS[1:]] + return np.array(accepted, dtype=dtype) + + +def counts(hosts, mass_range, zones, hubble): + """Population sizes the GUI displays alongside the candidate count. + + Returns a dict with the base-sample size and, per absolute-threshold zone, + the number of haloes above that zone's mass. + """ + low, high = mass_range + _, mvir, *_ = _physical(hosts, hubble) + result = {"base_sample": int(((mvir > low) & (mvir < high)).sum())} + for n, zone in enumerate(zones, start=1): + if not zone.is_relative: + result[f"above_zone{n}"] = int((mvir > zone.mass).sum()) + return result + + +def to_text(candidates): + """Serialise a selection to the ``candidates.dat`` format. + + One row per candidate, with the header the original writer emitted. + """ + lines = ["# ID M Rvir x y z"] + for row in candidates: + lines.append( + "%f %e %f %f %f %f " % ( + int(row["id"]), row["mvir"], row["rvir"], + row["posX"], row["posY"], row["posZ"], + ) + ) + return "\n".join(lines) + "\n" diff --git a/src/cme/ui/candidates.py b/src/cme/ui/candidates.py index d22cf58..d4be24f 100644 --- a/src/cme/ui/candidates.py +++ b/src/cme/ui/candidates.py @@ -1,6 +1,7 @@ from cme.ui.common import * -from cme import cosmology +from cme import cosmology, writers +from cme.science import candidates as sci from numpy.random import random class Candidates(HasTraits): @@ -123,151 +124,110 @@ def _haloid_default(self): #,140666,28221,147419,28188,147273,78411,131988,19910] def _addhalo_button_fired(self): - if len(self.candidatearr[:,0]) != 1: - if int(self.haloidtoadd) in self.candidatearr[:,0] and int(self.haloidtoadd) not in self.haloid: - self.haloid.append(int(self.haloidtoadd)) - self.jobstatus = "Candidate added." - elif int(self.haloidtoadd) in self.haloid: - self.jobstatus = "Candidate already in list!" - else: - self.jobstatus = "Candidate not found!" + # Previously `!= 1`, which tested for the phantom zero row that used to + # seed the results array. An empty selection now has no rows at all. + if len(self.candidatearr) == 0: + self.jobstatus = "You need to load some candidates, go back." + elif int(self.haloidtoadd) in self.haloid: + self.jobstatus = "Candidate already in list!" + elif int(self.haloidtoadd) in self.candidatearr[:, 0]: + self.haloid.append(int(self.haloidtoadd)) + self.jobstatus = "Candidate added." else: - self.jobstatus = "You need to load some candidates, go back." + self.jobstatus = "Candidate not found!" - self.main.mergertreetab.haloidlist = self.haloid + self._publish_sample() + + def _publish_sample(self): + """Push the working halo sample out to every tab that consumes it.""" self.main.halofindtab.haloidlist = self.haloid self.main.mergertreetab.haloidlist = self.haloid - self.main.initstab.haloid = self.haloid + self.main.contaminationtab.haloidlist = self.haloid self.main.gadgetruntab.haloidlist = self.haloid + self.main.initstab.haloid = self.haloid def _haloidtoadd_changed(self): self.jobstatus = "Waiting for next addition." def _clear_button_fired(self): + # Previously this set `mergertreetab.initstab`, a stray attribute on the + # wrong object, so the ICs tab kept the cleared sample. It also missed + # the contamination and Gadget tabs entirely. self.haloid = [] - self.main.halofindtab.haloidlist = [] - self.main.mergertreetab.haloidlist = [] - self.main.mergertreetab.initstab = [] + self._publish_sample() + + def _zones(self): + """The three exclusion zones as configured on this tab.""" + zones = [] + for distance, multiple, absolute, use_multiple in ( + (self.distancecut1, self.moremassivethancand1, + self.normlargerthan1, self.useintmult1), + (self.distancecut2, self.moremassivethancand2, + self.normlargerthan2, self.useintmult2), + (self.distancecut3, self.moremassivethancand3, + self.normlargerthan3, self.useintmult3), + ): + if use_multiple: + zones.append(sci.ExclusionZone( + distance=distance, mass_multiple=multiple)) + else: + zones.append(sci.ExclusionZone( + distance=distance, mass=absolute)) + return zones + + def _plot_candidates(self): + """Draw the current candidate positions on the shared figure.""" + if hasattr(self, 'display_points'): + figure = self.main.display + figure.clear() + figure.add_subplot(111) + + ax = self.main.display.axes[0] + self.display_points = ax.plot(self.candidatearr[:, 3], + self.candidatearr[:, 4], + marker=self.main.markerstyle, + linestyle='none', + markersize=self.main.markersize, + color=self.main.markercolor, + markeredgecolor=self.main.markercolor) + ax.set_xlim(0, 100) + ax.set_ylim(0, 100) + ax.set_xlabel('x-pos [Mpc/h]') + ax.set_ylabel('y-pos [Mpc/h]') + wx.CallAfter(self.main.display.canvas.draw) def _getcandidates_button_fired(self): hubble = cosmology.DEFAULT.h if self.loadpreviouscand == False: - halodata = RSDataReader.RSDataReader(self.halopath,self.snapnum,digits=2) - allhalos = halodata.get_hosts() - - MWcand = allhalos[np.logical_and(allhalos['mvir']/hubble>self.normlowermasscut, allhalos['mvir']/hubbleself.normlargerthan2] # HALOS LARGER THAN 7E12 - halos13 = allhalos[allhalos['mvir']/hubble>self.normlargerthan3] # HALOS LARGER THAN 7E13 - - self.nbetweenmasscut = len(MWcand) - self.nlargerthanlowermasscut = len(halos12) - self.nlargerthanuppermasscut = len(halos13) - - xpos12 = np.array(np.float64(halos12['posX'])) - ypos12 = np.array(np.float64(halos12['posY'])) - zpos12 = np.array(np.float64(halos12['posZ'])) - - xpos13 = np.array(np.float64(halos13['posX'])) - ypos13 = np.array(np.float64(halos13['posY'])) - zpos13 = np.array(np.float64(halos13['posZ'])) - - Ncandidates = 0 - if self.writeoutopt == True: - out = open(self.writeoutpath + self.writeoutname, 'w') - out.write('# ID M Rvir x y z\n') - - #xcand = [] - #ycand = [] - for i in range(0,len(MWcand)): - # CYCLE CANDIDATE i - xposi = np.array(np.float64(MWcand['posX']))[i] - yposi = np.array(np.float64(MWcand['posY']))[i] - zposi = np.array(np.float64(MWcand['posZ']))[i] - massi = np.array(np.float64(MWcand['mvir']))[i]/hubble - rviri = np.array(np.float64(MWcand['rvir']))[i] - idi = np.array(MWcand['id'])[i] - - # CALCULATE DISTANCE TO HALOS LARGER THAN 7E12 and 7E13 MSOL - - R12 = np.sqrt((xposi-xpos12)**2.+(yposi-ypos12)**2.+(zposi-zpos12)**2.)/hubble - R13 = np.sqrt((xposi-xpos13)**2.+(yposi-ypos13)**2.+(zposi-zpos13)**2.)/hubble - - # SELECT ALL HALOS LARGER THAN HALF THE SIZE OF THE CANDIDATE idi - - largerthanMW = allhalos[allhalos['mvir']/hubble >= self.moremassivethancand1*massi] - - # SINCE 0.5*massi INCLUDES idi, NEED TO REMOVE IT FROM X,Y,Z POS CALC SO MIN(R) != 0.0 - - idindex = np.where(largerthanMW['id'] != idi) - xtmp = np.array(np.float64(largerthanMW['posX'])) - ytmp = np.array(np.float64(largerthanMW['posY'])) - ztmp = np.array(np.float64(largerthanMW['posZ'])) - - xposMgtMW = xtmp[idindex[0]] - yposMgtMW = ytmp[idindex[0]] - zposMgtMW = ztmp[idindex[0]] - - RMgtMW = np.sqrt((xposi-xposMgtMW)**2.+(yposi-yposMgtMW)**2.+(zposi-zposMgtMW)**2.)/hubble - - # NO HALO LARGER THAN 7e13 CLOSER THAN 4 MPC - if R13.min() >= self.distancecut3: - # NO HALO LARGER THAN 7e12 CLOSER THAN 3 MPC - if R12.min() >= self.distancecut2: - # NO HALO HALF THE MASS OF CANDIDATE OR LARGER WITHIN 1.4 MPC - if RMgtMW.min() >= self.distancecut1: - self.candidatearr = np.vstack([self.candidatearr, [idi,massi,rviri,xposi,yposi,zposi]]) - Ncandidates += 1 - if self.writeoutopt == True: - out.write('%f %e %f %f %f %f \n' % - (int(idi),massi,rviri,xposi,yposi,zposi)) - - self.ntotalcandidates = Ncandidates - if Ncandidates > 0: - if hasattr(self, 'display_points'): - figure = self.main.display - figure.clear() - ax = figure.add_subplot(111) - - ax = self.main.display.axes[0] - - self.display_points = ax.plot(self.candidatearr[:,3],self.candidatearr[:,4],marker=self.main.markerstyle,linestyle='none',markersize=self.main.markersize,color=self.main.markercolor,markeredgecolor=self.main.markercolor) - ax.set_xlim(0,100) - ax.set_ylim(0,100) - ax.set_xlabel('posX') - ax.set_ylabel('posY') - wx.CallAfter(self.main.display.canvas.draw) - + halodata = RSDataReader.RSDataReader(self.halopath, self.snapnum, digits=2) + hosts = halodata.get_hosts() + + mass_range = (self.normlowermasscut, self.normuppermasscut) + zones = self._zones() + + tallies = sci.counts(hosts, mass_range, zones, hubble) + self.nbetweenmasscut = tallies['base_sample'] + self.nlargerthanlowermasscut = tallies.get('above_zone2', 0) + self.nlargerthanuppermasscut = tallies.get('above_zone3', 0) + + found = sci.select(hosts, mass_range, zones, hubble) + self.ntotalcandidates = len(found) + self.candidatearr = np.column_stack( + [found[name] for name in sci.COLUMNS] + ) if len(found) else np.zeros(shape=(0, 6)) + if self.writeoutopt == True: - out.close() + writers.write(os.path.join(self.writeoutpath, self.writeoutname), + sci.to_text(found)) + + if len(found): + self._plot_candidates() elif self.loadpreviouscand == True: - cand = getcandidatelist(self.writeoutpath + self.writeoutname) - self.candidatearr = np.zeros(shape=(len(cand[:,0]),6)) - self.candidatearr[:,0] = cand[:,0] - self.candidatearr[:,1] = cand[:,1] - self.candidatearr[:,2] = cand[:,2] - self.candidatearr[:,3] = cand[:,3] - self.candidatearr[:,4] = cand[:,4] - self.candidatearr[:,5] = cand[:,5] - self.ntotalcandidates = len(cand[:,0]) - if hasattr(self, 'display_points'): - figure = self.main.display - figure.clear() - ax = figure.add_subplot(111) - - ax = self.main.display.axes[0] - self.display_points = ax.plot(self.candidatearr[:,3],self.candidatearr[:,4], - marker=self.main.markerstyle, - linestyle='none', - markersize=self.main.markersize, - color=self.main.markercolor, - markeredgecolor=self.main.markercolor) - ax.set_xlim(0,100) - ax.set_ylim(0,100) - ax.set_xlabel('x-pos [Mpc/h]') - ax.set_ylabel('y-pos [Mpc/h]') - wx.CallAfter(self.main.display.canvas.draw) + cand = getcandidatelist(os.path.join(self.writeoutpath, self.writeoutname)) + self.candidatearr = np.asarray(cand)[:, :6] + self.ntotalcandidates = len(self.candidatearr) + self._plot_candidates() def _largerthan1_changed(self): self.normlargerthan1 = 10**self.largerthan1 @@ -313,7 +273,7 @@ def __init__(self, main, **kwargs): self.normlowermasscut = 10**self.lowermasscut self.normuppermasscut = 10**self.uppermasscut - self.candidatearr = np.zeros(shape=(1,6)) + self.candidatearr = np.zeros(shape=(0, 6)) self.distancecut1 = 1.4 self.distancecut2 = 3. diff --git a/tests/test_science_candidates.py b/tests/test_science_candidates.py new file mode 100644 index 0000000..93991a0 --- /dev/null +++ b/tests/test_science_candidates.py @@ -0,0 +1,257 @@ +import numpy as np +import pytest + +from cme.science import candidates +from cme.science.candidates import ExclusionZone + + +def catalogue(rows): + """Build a host catalogue from (id, mvir, rvir, x, y, z) tuples.""" + keys = ("id", "mvir", "rvir", "posX", "posY", "posZ") + return {k: [row[i] for row in rows] for i, k in enumerate(keys)} + + +MW = 1e12 # comfortably inside the default 7e11 - 7e12 base sample +RANGE = (7e11, 7e12) + + +class TestExclusionZone: + def test_requires_exactly_one_mass_specification(self): + with pytest.raises(ValueError, match="exactly one"): + ExclusionZone(distance=1.0) + with pytest.raises(ValueError, match="exactly one"): + ExclusionZone(distance=1.0, mass=1e12, mass_multiple=0.5) + + def test_is_relative_reflects_which_was_given(self): + assert ExclusionZone(distance=1.0, mass_multiple=0.5).is_relative + assert not ExclusionZone(distance=1.0, mass=1e12).is_relative + + +class TestMassRange: + def test_selects_only_haloes_inside_the_range(self): + cat = catalogue([ + (1, 1e11, 100.0, 10.0, 10.0, 10.0), # too light + (2, MW, 200.0, 30.0, 30.0, 30.0), # in range + (3, 1e14, 900.0, 50.0, 50.0, 50.0), # too heavy + ]) + out = candidates.select(cat, RANGE, [], hubble=1.0) + assert list(out["id"]) == [2] + + def test_range_is_exclusive_at_both_ends(self): + cat = catalogue([ + (1, 7e11, 100.0, 10.0, 10.0, 10.0), + (2, 7e12, 100.0, 30.0, 30.0, 30.0), + ]) + assert len(candidates.select(cat, RANGE, [], hubble=1.0)) == 0 + + def test_returns_an_empty_array_not_a_zero_row(self): + # AUDIT 2.2.3: the original seeded its results with np.zeros((1, 6)), so + # every selection carried a phantom zero-mass candidate at the origin. + cat = catalogue([(1, 1e14, 900.0, 10.0, 10.0, 10.0)]) + out = candidates.select(cat, RANGE, [], hubble=1.0) + assert len(out) == 0 + assert out["mvir"].sum() == 0.0 # because there are no rows at all + assert 0 not in list(out["id"]) + + +class TestHubbleConversion: + def test_masses_are_converted_before_the_range_test(self): + # 5e12 Msol/h is 1e13 Msol at h=0.5, which is outside the range. + cat = catalogue([(1, 5e12, 200.0, 10.0, 10.0, 10.0)]) + assert len(candidates.select(cat, RANGE, [], hubble=1.0)) == 1 + assert len(candidates.select(cat, RANGE, [], hubble=0.5)) == 0 + + def test_distances_are_converted_before_the_zone_test(self): + # Separation is 2 Mpc/h. At h=1.0 that is 2 Mpc and clears a 1.5 Mpc + # zone; at h=0.5 it is 4 Mpc and clears it by more. + cat = catalogue([ + (1, MW, 200.0, 10.0, 10.0, 10.0), + (2, 1e14, 900.0, 12.0, 10.0, 10.0), + ]) + zone = [ExclusionZone(distance=3.0, mass=1e13)] + assert len(candidates.select(cat, RANGE, zone, hubble=1.0)) == 0 + assert len(candidates.select(cat, RANGE, zone, hubble=0.5)) == 1 + + +class TestAbsoluteExclusionZones: + def test_rejects_a_candidate_with_a_massive_neighbour_inside_the_zone(self): + cat = catalogue([ + (1, MW, 200.0, 10.0, 10.0, 10.0), + (2, 1e14, 900.0, 12.0, 10.0, 10.0), # 2 Mpc away + ]) + zones = [ExclusionZone(distance=4.0, mass=7e13)] + assert len(candidates.select(cat, RANGE, zones, hubble=1.0)) == 0 + + def test_accepts_when_the_neighbour_is_beyond_the_zone(self): + cat = catalogue([ + (1, MW, 200.0, 10.0, 10.0, 10.0), + (2, 1e14, 900.0, 20.0, 10.0, 10.0), # 10 Mpc away + ]) + zones = [ExclusionZone(distance=4.0, mass=7e13)] + assert list(candidates.select(cat, RANGE, zones, hubble=1.0)["id"]) == [1] + + def test_accepts_when_the_neighbour_is_below_the_mass_threshold(self): + cat = catalogue([ + (1, MW, 200.0, 10.0, 10.0, 10.0), + # Close, but far below the zone's mass -- and below the base sample + # range too, so it is not itself a candidate. + (2, 1e11, 90.0, 11.0, 10.0, 10.0), + ]) + zones = [ExclusionZone(distance=4.0, mass=7e13)] + assert list(candidates.select(cat, RANGE, zones, hubble=1.0)["id"]) == [1] + + def test_all_zones_must_pass(self): + cat = catalogue([ + (1, MW, 200.0, 10.0, 10.0, 10.0), + (2, 8e12, 300.0, 12.0, 10.0, 10.0), # violates the near zone only + ]) + near = ExclusionZone(distance=3.0, mass=7e12) + far = ExclusionZone(distance=4.0, mass=7e13) + assert len(candidates.select(cat, RANGE, [far], hubble=1.0)) == 1 + assert len(candidates.select(cat, RANGE, [near, far], hubble=1.0)) == 0 + + def test_distance_is_measured_in_three_dimensions(self): + # (3, 4, 0) is 5 Mpc away, not 3 or 4. + cat = catalogue([ + (1, MW, 200.0, 10.0, 10.0, 10.0), + (2, 1e14, 900.0, 13.0, 14.0, 10.0), + ]) + assert len(candidates.select( + cat, RANGE, [ExclusionZone(distance=4.9, mass=7e13)], hubble=1.0)) == 1 + assert len(candidates.select( + cat, RANGE, [ExclusionZone(distance=5.1, mass=7e13)], hubble=1.0)) == 0 + + +class TestRelativeExclusionZones: + def test_excludes_the_candidate_itself(self): + # A halo is zero distance from itself and always satisfies + # "at least half my mass", so without self-exclusion nothing would ever + # be selected. + cat = catalogue([(1, MW, 200.0, 10.0, 10.0, 10.0)]) + zones = [ExclusionZone(distance=1.4, mass_multiple=0.5)] + assert list(candidates.select(cat, RANGE, zones, hubble=1.0)["id"]) == [1] + + def test_rejects_a_comparable_mass_neighbour(self): + cat = catalogue([ + (1, MW, 200.0, 10.0, 10.0, 10.0), + (2, 0.8 * MW, 190.0, 11.0, 10.0, 10.0), # 1 Mpc, > half the mass + ]) + zones = [ExclusionZone(distance=1.4, mass_multiple=0.5)] + assert len(candidates.select(cat, RANGE, zones, hubble=1.0)) == 0 + + def test_ignores_a_much_lighter_neighbour(self): + cat = catalogue([ + (1, MW, 200.0, 10.0, 10.0, 10.0), + (2, 0.1 * MW, 90.0, 11.0, 10.0, 10.0), # 1 Mpc, well under half + ]) + zones = [ExclusionZone(distance=1.4, mass_multiple=0.5)] + assert list(candidates.select(cat, RANGE, zones, hubble=1.0)["id"]) == [1] + + def test_threshold_scales_with_each_candidate(self): + # The 2e12 halo is more than half of 3e12 but the 1e12 halo is not, + # so only the heavier candidate is rejected by its neighbour. + cat = catalogue([ + (1, 1e12, 200.0, 10.0, 10.0, 10.0), + (2, 3e12, 300.0, 40.0, 40.0, 40.0), + (3, 2e12, 250.0, 41.0, 40.0, 40.0), + ]) + zones = [ExclusionZone(distance=1.4, mass_multiple=0.5)] + out = candidates.select(cat, RANGE, zones, hubble=1.0) + assert list(out["id"]) == [1] + + +class TestDefaultCaterpillarCriteria: + """The criteria the GUI shipped as defaults, as one integration case.""" + + ZONES = [ + ExclusionZone(distance=1.4, mass_multiple=0.5), + ExclusionZone(distance=3.0, mass=7e12), + ExclusionZone(distance=4.0, mass=7e13), + ] + + def test_an_isolated_milky_way_analogue_is_selected(self): + cat = catalogue([ + (100, 1.5e12, 250.0, 50.0, 50.0, 50.0), + (200, 1e14, 900.0, 70.0, 50.0, 50.0), # 20 Mpc: clears all zones + ]) + out = candidates.select(cat, RANGE, self.ZONES, hubble=1.0) + assert list(out["id"]) == [100] + + def test_a_cluster_neighbour_at_3_mpc_disqualifies(self): + cat = catalogue([ + (100, 1.5e12, 250.0, 50.0, 50.0, 50.0), + (200, 1e14, 900.0, 53.0, 50.0, 50.0), # inside the 4 Mpc zone + ]) + assert len(candidates.select(cat, RANGE, self.ZONES, hubble=1.0)) == 0 + + def test_returns_columns_in_the_documented_order(self): + cat = catalogue([(100, 1.5e12, 250.0, 50.0, 51.0, 52.0)]) + out = candidates.select(cat, RANGE, self.ZONES, hubble=1.0) + assert out.dtype.names == candidates.COLUMNS + assert out[0]["posY"] == 51.0 + assert out[0]["rvir"] == 250.0 + + def test_rvir_is_not_hubble_scaled(self): + # rvir passes through in kpc/h as Rockstar reports it; only mvir and + # the distances are converted. + cat = catalogue([(100, 1.5e12, 250.0, 50.0, 50.0, 50.0)]) + out = candidates.select(cat, RANGE, [], hubble=0.6711) + assert out[0]["rvir"] == 250.0 + + +class TestCounts: + def test_reports_base_sample_and_per_zone_populations(self): + cat = catalogue([ + (1, 1e12, 200.0, 10.0, 10.0, 10.0), + (2, 2e12, 250.0, 20.0, 20.0, 20.0), + (3, 1e13, 500.0, 30.0, 30.0, 30.0), + (4, 1e14, 900.0, 40.0, 40.0, 40.0), + ]) + zones = [ + ExclusionZone(distance=1.4, mass_multiple=0.5), + ExclusionZone(distance=3.0, mass=7e12), + ExclusionZone(distance=4.0, mass=7e13), + ] + result = candidates.counts(cat, RANGE, zones, hubble=1.0) + assert result["base_sample"] == 2 # the 1e12 and 2e12 haloes + assert result["above_zone2"] == 2 # 1e13 and 1e14 + assert result["above_zone3"] == 1 # 1e14 only + assert "above_zone1" not in result # relative zones have no fixed count + + +class TestSerialisation: + def test_round_trips_through_the_candidates_dat_format(self): + cat = catalogue([(190897, 1.5e12, 250.0, 50.1, 51.2, 52.3)]) + out = candidates.select(cat, RANGE, [], hubble=1.0) + text = candidates.to_text(out) + + assert text.startswith("# ID M Rvir x y z\n") + values = text.splitlines()[1].split() + assert int(float(values[0])) == 190897 + assert float(values[1]) == pytest.approx(1.5e12) + assert float(values[3]) == pytest.approx(50.1) + + def test_empty_selection_writes_only_a_header(self): + empty = candidates.select(catalogue([]), RANGE, [], hubble=1.0) + assert candidates.to_text(empty) == "# ID M Rvir x y z\n" + + +class TestScaling: + def test_handles_a_large_catalogue_without_quadratic_extraction(self): + # AUDIT 4.5: the original re-extracted six full arrays from the + # catalogue inside the per-candidate loop. This is a smoke test that a + # realistic base sample completes promptly. + rng = np.random.default_rng(0) + n = 4000 + cat = { + "id": np.arange(n), + "mvir": rng.uniform(7.1e11, 6.9e12, n), + "rvir": rng.uniform(100.0, 300.0, n), + "posX": rng.uniform(0.0, 100.0, n), + "posY": rng.uniform(0.0, 100.0, n), + "posZ": rng.uniform(0.0, 100.0, n), + } + out = candidates.select( + cat, RANGE, [ExclusionZone(distance=1.4, mass_multiple=0.5)], hubble=0.6711 + ) + assert 0 < len(out) <= n From 636b0ad9325b8d6a334a190eb19b23f64d1300cd Mon Sep 17 00:00:00 2001 From: Brendan Griffen Date: Mon, 27 Jul 2026 22:20:52 +1000 Subject: [PATCH 11/19] perf: hoist loop-invariant parent snapshot reads out of the nrvir loop The Lagrangian region builder re-read four large blocks from the parent simulation on every iteration of the nrvir loop, none of which depend on nrvir: the snapshot positions and IDs, and the IC positions and IDs. For a 512^3 parent each POS block is roughly 1.6 GB, so a sweep over five region sizes moved about 30 GB from disk to compute the same four arrays five times. Read them once before the loop. The particle radii from the candidate centre are invariant too -- only the acceptance threshold moves with nrvir -- so R is computed once as well, and the position block it derives from is freed immediately afterwards. Peak memory is no worse. The original held the (N,3) position block alongside three (N,) difference arrays and R; this holds the (N,3) IC position block alongside three (N,) index and ID arrays, and it never materialises dx/dy/dz as separate arrays. Also drops `currentpos = snapPOS[Rindex[0]]`, a large allocation whose result was never read, and replaces three copies of the hardcoded parent snapshot path with module constants noting that they belong in configuration. --- src/cme/ui/ics.py | 53 ++++++++++++++++++++++++++++------------------- 1 file changed, 32 insertions(+), 21 deletions(-) diff --git a/src/cme/ui/ics.py b/src/cme/ui/ics.py index 108013c..b069d89 100644 --- a/src/cme/ui/ics.py +++ b/src/cme/ui/ics.py @@ -5,6 +5,12 @@ #: Side length of the Caterpillar parent box, in Mpc/h. PARENT_BOXLENGTH = 100 + +#: Paths to the parent simulation's final snapshot and its initial +#: conditions, relative to the parent simulation directory. Hardcoded to +#: the Caterpillar 512^3 parent; these belong in configuration. +PARENT_SNAPSHOT = "/512Parent/outputs/snapdir_063/snap_063" +PARENT_ICS = "/512Parent/ics/ics" from glob import glob import random @@ -575,8 +581,8 @@ def _makeic_button_fired(self): posYcand = allhalos.loc[idhalo]['posY'] posZcand = allhalos.loc[idhalo]['posZ'] - ext = "/512Parent/outputs/snapdir_063/snap_063" - header=rsHD.snapshot_header(self.main.headertab.parentsimpath+ext) + header = rsHD.snapshot_header( + self.main.headertab.parentsimpath + PARENT_SNAPSHOT) print("------------------------------------------------") print("Rockstar ID inside parent simulation: ",idhalo) @@ -596,31 +602,36 @@ def _makeic_button_fired(self): self.halomvir = '{0:.2e}'.format(float(allhalos.loc[idhalo]['mvir'])/header.hubble) self.halorvir = '{:.2f}'.format(float(allhalos.loc[idhalo]['rvir'])) + # None of these four blocks depends on Nrvir, so read them once + # rather than once per requested region size. Each POS block is + # ~1.6 GB for a 512^3 parent, so re-reading them per iteration + # dominated the runtime of a multi-nrvir sweep. + snappath = self.main.headertab.parentsimpath + PARENT_SNAPSHOT + icspath = self.main.headertab.parentsimpath + PARENT_ICS + + snapPOS = rsHD.read_block(snappath, "POS ") + # The radius of every parent particle from the candidate centre + # is invariant too -- only the threshold moves with Nrvir. Freed + # immediately, since it is the largest array in play. + R = np.sqrt( + (allhalos.loc[idhalo]['posX'] - snapPOS[:, 0])**2. + + (allhalos.loc[idhalo]['posY'] - snapPOS[:, 1])**2. + + (allhalos.loc[idhalo]['posZ'] - snapPOS[:, 2])**2.) + del snapPOS + + snapIDS = rsHD.read_block(snappath, "ID ") + snapIDSlagr = rs.read_block(icspath, "ID ", doubleprec=False) + snapPOSlagr = rs.read_block(icspath, "POS ", doubleprec=False) + for Nrvir in self.nrvir: print('Constructing: H' + str(self.haloidselect) + 'NRVIR' + str(int(Nrvir))) Nrvir = float(Nrvir) - ext = "/512Parent/outputs/snapdir_063/snap_063" - snapPOS = rsHD.read_block(self.main.headertab.parentsimpath+ext,"POS ") - dx = allhalos.loc[idhalo]['posX'] - snapPOS[:,0] - dy = allhalos.loc[idhalo]['posY'] - snapPOS[:,1] - dz = allhalos.loc[idhalo]['posZ'] - snapPOS[:,2] - R = np.sqrt(dx**2. + dy**2. + dz**2.) Rindex = np.where(R < Nrvir*rvircand/1000) - currentpos = snapPOS[Rindex[0]] - del snapPOS, dx, dy, dz - - snapIDS=rsHD.read_block(self.main.headertab.parentsimpath+ext,"ID ") regionIDS = snapIDS[Rindex[0]] - del snapIDS - - extics = "/512Parent/ics/ics" - snapIDSlagr = rs.read_block(self.main.headertab.parentsimpath+extics,"ID ",doubleprec=False) - mask = np.in1d(snapIDSlagr,regionIDS,assume_unique=True) - del snapIDSlagr - snapPOSlagr = rs.read_block(self.main.headertab.parentsimpath+extics,"POS ",doubleprec=False) - lagrPos=snapPOSlagr[mask] + mask = np.in1d(snapIDSlagr, regionIDS, assume_unique=True) + lagrPos = snapPOSlagr[mask] del mask, regionIDS - + CorrectPos(lagrPos[:,0],header.boxsize) CorrectPos(lagrPos[:,1],header.boxsize) CorrectPos(lagrPos[:,2],header.boxsize) From 0f30c6b8246a79dac1ca4f2b79f9933e802943ab Mon Sep 17 00:00:00 2001 From: Brendan Griffen Date: Mon, 27 Jul 2026 22:22:31 +1000 Subject: [PATCH 12/19] fix: correct typo'd trait assignments that silently disabled features Traits accepts assignment to an undeclared attribute by creating it, so a misspelled trait name fails silently: the real trait keeps its default and the feature it controls quietly does nothing. Four instances: - AUDIT 2.1.2, gadget.py: `self.nvir = ['3',...]` where the trait is `nrvir`. nrvir stayed an empty list, so every `for nrviri in self.nrvir` loop body was skipped and the Gadget parameter sweep produced nothing at all. This is the most consequential of the four -- the tab's main action was inert. - AUDIT 2.1.7, home.py: `self.clustopt = 'odyssey'` where the trait is `clusteropt`, so the Harvard Odyssey preset never applied and the cluster stayed at its default. - AUDIT 2.1.9, ics.py: `self.pre_smooth` / `self.post_smooth` / `self.grad_order` where the traits are `presmooth` / `postsmooth` / `gradorder`. Three MUSIC Poisson-solver settings were never applied. Compounding it, `presmooth` and `postsmooth` were declared as plain class integers rather than traits, while the view built Item()s for them; they are now Int traits. Fixing the grad_order typo made an existing correct assignment redundant, so the duplicate is removed. --- src/cme/ui/gadget.py | 2 +- src/cme/ui/home.py | 2 +- src/cme/ui/ics.py | 11 +++++------ 3 files changed, 7 insertions(+), 8 deletions(-) diff --git a/src/cme/ui/gadget.py b/src/cme/ui/gadget.py index d5f2295..7d77a08 100644 --- a/src/cme/ui/gadget.py +++ b/src/cme/ui/gadget.py @@ -629,7 +629,7 @@ def __init__(self, main, **kwargs): HasTraits.__init__(self) self.main = main self.boxtype = ['ellipsoid'] - self.nvir = ['3','4','5','6','7','8','9'] + self.nrvir = ['3','4','5','6','7','8','9'] self.padding = ['7','8','9'] self.overlap = ['4'] self.lmax = '11' diff --git a/src/cme/ui/home.py b/src/cme/ui/home.py index e68758c..0abf7b6 100644 --- a/src/cme/ui/home.py +++ b/src/cme/ui/home.py @@ -72,7 +72,7 @@ def __init__(self, main, **kwargs): self.username = 'uqbgriff' if platform.node() == 'rclogin13.rc.fas.harvard.edu': - self.clustopt = 'odyssey' + self.clusteropt = 'odyssey' self.homepath = '/n/home01/bgriffen/data/' self.masterpath = self.homepath + self.username self.gadgetpath = self.masterpath + '/lib/P-Gadget3' diff --git a/src/cme/ui/ics.py b/src/cme/ui/ics.py index b069d89..cf3c295 100644 --- a/src/cme/ui/ics.py +++ b/src/cme/ui/ics.py @@ -83,8 +83,8 @@ class InitialConditions(HasTraits): #POISSON fftfine = Bool(False) accuracy = Float - presmooth = 3 - postsmooth = 3 + presmooth = Int(3) + postsmooth = Int(3) smoother = Str laplaceorder = Int gradorder = Int @@ -743,14 +743,14 @@ def __init__(self, main, **kwargs): self.accuracy = 0.00001 self.fftfine = True - self.pre_smooth = 3 - self.post_smooth = 3 + self.presmooth = 3 + self.postsmooth = 3 self.smoother = 'gs' self.projopt = 'xy' self.tranfunc = ['eisenstein'] self.cosmologylist = ['PLANCK'] self.laplaceorder = 6 - self.grad_order = 6 + self.gradorder = 6 self.lagroutput = str(self.main.headertab.masterpath) + '/' + str(self.toplagr) + '/H' + str(self.haloidselect) + 'NRVIR' + str(self.nrvir) self.resimlagrdir = str(self.main.headertab.datamasterpath) self.resimlagrfile = '/' + str(self.toplagr) + '/H' + str(self.haloidselect) + 'NRVIR' + str(self.nrvir) @@ -767,7 +767,6 @@ def __init__(self, main, **kwargs): self.padding = ['7'] self.overlap = ['4'] self.lmax = ['10'] - self.gradorder = 6 self.noutput = 4 #self.parentsimconf = 'ics_example.conf' self.parentsimpath = self.main.headertab.parentsimpath + '512Parent/ics/' From 4010574c30460dfce6ce4f2d23747a75a9c572ee Mon Sep 17 00:00:00 2001 From: Brendan Griffen Date: Mon, 27 Jul 2026 22:29:16 +1000 Subject: [PATCH 13/19] fix: correct wrong coordinates, unbound locals and dead control flow Seven defects across the halo and ICs tabs. Wrong results: - AUDIT 2.2.1, halos.py: the 3D halo distribution assigned `z = tmphalos['posY']`, so every full-box Mayavi scene was flattened into a plane with the y coordinate plotted twice. - AUDIT 2.2.2, halos.py: `self.hostposx = host['posZ']`, so the reported host x position was actually its z position. - AUDIT 2.2.4, ics.py: the displayed Lagrangian pointer name and path formatted the nrvir CheckListEditor list with str(), producing "NRVIR['1']" -- visible in screenshots/constructICs.png -- while the files were written with int(nrvir[0]). Displayed and actual paths disagreed. Both now go through naming.lagr_region_path, and the labels clear when the selection is ambiguous. - ics.py: `for lmini in self.lmin` iterated an Enum holding a single string, so a levelmin of 10 or above walked its characters and ran the sweep twice with levelmin '1' and then '0'. lmin is not a list like the other sweep axes. Crashes and dead code paths: - AUDIT 2.1.4, halos.py: gethostid() returned an unbound local when no row matched, raising UnboundLocalError, even though every caller tests the result for truthiness as though a miss were expected. It now returns None, tolerates a missing summary file, and skips short rows. - AUDIT 2.1.5, halos.py: gethalos_xy() read `idhost` in its radius block, but only the zoom branch defines it, so every parent full-box plot with two position axes raised NameError. It also returned unbound `x, y` when the catalogue was missing. The method is restructured with the catalogue load and halo-type selection pulled out, and returns empty arrays on a miss. - AUDIT 2.1.6, halos.py: the entire plotting block in _plot_button_fired sat inside the `elif parentorzoom == 'zoom'` branch, so pressing Plot in parent mode computed a path and then fell off the end of the method drawing nothing. - AUDIT 2.1.10, ics.py: `nhalo` was unbound when the selected halo was absent from the candidate file. Replaced with a vectorised lookup that reports the miss through the status field. Also replaces the hardcoded /bigbang absolute path in gethostid() with a summarypath trait derived from the project data directory, and renames the `reWriteIC` import from `re` -- which shadowed the standard library module -- to `rewriteic`. --- src/cme/ui/halos.py | 284 ++++++++++++++++++++++++-------------------- src/cme/ui/ics.py | 86 +++++++++----- 2 files changed, 206 insertions(+), 164 deletions(-) diff --git a/src/cme/ui/halos.py b/src/cme/ui/halos.py index c0f49f9..93bec42 100644 --- a/src/cme/ui/halos.py +++ b/src/cme/ui/halos.py @@ -5,6 +5,7 @@ class HaloFind(HasTraits): halopath = Directory + summarypath = Directory x_scale = Enum(['linear','log']) y_scale = Enum(['linear','log']) @@ -139,145 +140,162 @@ def _run_kwargs(self): nrvir=self.nrvir) def gethostid(self): - path = "/bigbang/data/AnnaGroup/caterpillar/halos" - with open(path + "/halosummary.txt") as f: - for line in f: - if line[0] != '#': - linecomp = line.split() - if 'H'+str(self.haloid) == linecomp[0] and 'B'+str(self.boxtype.upper())[0] == linecomp[1] \ - and str(self.lmax) == linecomp[2] and str(self.nrvir) == linecomp[3]: - haloid = linecomp[17] + """The Rockstar ID of this zoom's host halo, or None if not listed. - return haloid + Previously this returned an unbound local when no row matched, raising + UnboundLocalError, even though every caller tests the result for + truthiness as though a miss were expected (AUDIT 2.1.4). + """ + summary = os.path.join(self.summarypath, "halosummary.txt") + if not os.path.exists(summary): + return None - def gethalos_xy(self): + with open(summary) as f: + for line in f: + if line.startswith('#'): + continue + linecomp = line.split() + if len(linecomp) < 18: + continue + if ('H' + str(self.haloid) == linecomp[0] + and 'B' + naming.region_code(self.boxtype) == linecomp[1] + and str(self.lmax) == linecomp[2] + and str(self.nrvir) == linecomp[3]): + return linecomp[17] + + return None + + def select_halo_type(self, halodata): + """The subset of a catalogue matching the selected halo type.""" + if self.halo_type == 'hosts only': + return halodata.get_hosts() + if self.halo_type == 'subhalos only': + return halodata.get_subs() + return halodata.data + + def catalogue_path(self): + """Path to the Rockstar catalogue for whichever mode is selected.""" if self.parentorzoom == 'parent': - self.halopath = self.main.headertab.parentsimpath + 'RockstarData' - if os.path.exists(self.halopath): - halodata = RSDataReader.RSDataReader(self.halopath,self.snapshot,digits=2) - self.datastatus = 'Parent found.' - dataexists = True - else: - dataexists = False - self.datastatus = 'Parent not found.' - - elif self.parentorzoom == 'zoom': - self.foldername = naming.run_dir_name(halo_id=self.haloid, **self._run_kwargs()) - self.halopath = naming.rockstar_dir(self.gadpath, self.haloid, **self._run_kwargs()) - - if os.path.exists(self.halopath): - halodata = RSDataReader.RSDataReader(self.halopath,self.snapshot,digits=2) - self.datastatus = 'Zoom found.' - dataexists = True - else: - dataexists = False - self.datastatus = 'Zoom not found.' - - if dataexists: - if self.halo_type == 'all halos': - tmphalos = halodata.data - if self.halo_type == 'hosts only': - tmphalos = halodata.get_hosts() - if self.halo_type == 'subhalos only': - tmphalos = halodata.get_subs() - if self.halo_type == 'subs of ID': - tmphalos = halodata.get_subhalos_from_halo(self.haloid) + return os.path.join(self.main.headertab.parentsimpath, 'RockstarData') + self.foldername = naming.run_dir_name(halo_id=self.haloid, **self._run_kwargs()) + return naming.rockstar_dir(self.gadpath, self.haloid, **self._run_kwargs()) + + def load_catalogue(self): + """Read the selected catalogue, or return None if it is not on disk.""" + self.halopath = self.catalogue_path() + label = 'Parent' if self.parentorzoom == 'parent' else 'Zoom' + if not os.path.exists(self.halopath): + self.datastatus = f'{label} not found.' + return None + self.datastatus = f'{label} found.' + return RSDataReader.RSDataReader(self.halopath, self.snapshot, digits=2) - - if self.fullboxopt == 'full box' and self.parentorzoom == 'parent': - x = np.array(tmphalos[self.halo_varx]) - y = np.array(tmphalos[self.halo_vary]) - elif self.parentorzoom == 'zoom': - idhost = self.gethostid() - if idhost: - hosts = halodata.get_hosts() - subs = halodata.get_subs() - host = hosts.loc[int(idhost)] - subsinhost = halodata.get_subhalos_from_halo(self.haloid) - xposhost = float(host['posX']) - yposhost = float(host['posY']) - zposhost = float(host['posZ']) - self.zoomid = int(idhost) - self.hostmvir = host['mvir']/halodata.h0 - self.hostrvir = host['rvir'] - self.hostrvmax = host['rvmax'] - self.hostposx = host['posZ'] - self.hostposy = host['posY'] - self.hostposz = host['posZ'] - self.nhosts = len(hosts['posX']) - self.nsubs = len(subs['posX']) - self.sinhost = len(subsinhost['posX']) - cond1h = (np.array(tmphalos['posX']) >= xposhost - self.deltar/2.) & (np.array(tmphalos['posX']) <= xposhost + self.deltar/2.) - cond2h = (np.array(tmphalos['posY']) >= yposhost - self.deltar/2.) & (np.array(tmphalos['posY']) <= yposhost + self.deltar/2.) - cond3h = (np.array(tmphalos['posZ']) >= zposhost - self.deltar/2.) & (np.array(tmphalos['posZ']) <= zposhost + self.deltar/2.) - condh = cond1h & cond2h & cond3h - - x = np.array(tmphalos[self.halo_varx][condh]) - y = np.array(tmphalos[self.halo_vary][condh]) - self.datastatus = 'Host found.' - elif not idhost: - self.datastatus = "No host found, plotting all." - x = np.array(tmphalos[self.halo_varx]) - y = np.array(tmphalos[self.halo_vary]) - - if 'pos' in self.halo_varx and 'pos' in self.halo_vary: - if idhost: - rvir = np.array(tmphalos['rvir'][condh]) - elif not idhost: - rvir = np.array(tmphalos['rvir']) - - x,y = drawcircle(x,y,rvir/1000) - + def gethalos_xy(self): + """The selected x and y quantities, restricted to the region of interest. + + Returns empty arrays when the catalogue is missing. Previously the + function fell through to `return x, y` with both names unbound + (AUDIT 2.1.5). + """ + halodata = self.load_catalogue() + if halodata is None: + return np.array([]), np.array([]) + + tmphalos = self.select_halo_type(halodata) + + # Only the zoom branch establishes a host. The radius block below used + # to read `idhost` unconditionally, so every parent full-box plot with + # two position axes raised NameError (AUDIT 2.1.5). + idhost = None + condh = None + + if self.parentorzoom == 'zoom': + idhost = self.gethostid() + + if idhost: + hosts = halodata.get_hosts() + subs = halodata.get_subs() + host = hosts.loc[int(idhost)] + subsinhost = halodata.get_subhalos_from_halo(self.haloid) + xposhost = float(host['posX']) + yposhost = float(host['posY']) + zposhost = float(host['posZ']) + self.zoomid = int(idhost) + self.hostmvir = host['mvir']/halodata.h0 + self.hostrvir = host['rvir'] + self.hostrvmax = host['rvmax'] + self.hostposx = host['posX'] + self.hostposy = host['posY'] + self.hostposz = host['posZ'] + self.nhosts = len(hosts['posX']) + self.nsubs = len(subs['posX']) + self.sinhost = len(subsinhost['posX']) + + half = self.deltar/2. + condh = np.ones(len(tmphalos['posX']), dtype=bool) + for axis, centre in (('posX', xposhost), + ('posY', yposhost), + ('posZ', zposhost)): + values = np.array(tmphalos[axis]) + condh &= (values >= centre - half) & (values <= centre + half) + + x = np.array(tmphalos[self.halo_varx][condh]) + y = np.array(tmphalos[self.halo_vary][condh]) + self.datastatus = 'Host found.' else: - self.datastatus = 'Data not found or plotted.' + if self.parentorzoom == 'zoom': + self.datastatus = "No host found, plotting all." + x = np.array(tmphalos[self.halo_varx]) + y = np.array(tmphalos[self.halo_vary]) - return x,y - + if 'pos' in self.halo_varx and 'pos' in self.halo_vary: + rvir = np.array(tmphalos['rvir'] if condh is None + else tmphalos['rvir'][condh]) + x, y = drawcircle(x, y, rvir/1000) - def _plot_button_fired(self): - if self.parentorzoom == 'parent': - self.halopath = self.main.headertab.parentsimpath + 'RockstarData' - if os.path.exists(self.halopath): - dataexists = True - else: - dataexists = False - - elif self.parentorzoom == 'zoom': - self.foldername = naming.run_dir_name(halo_id=self.haloid, **self._run_kwargs()) - self.halopath = naming.rockstar_dir(self.gadpath, self.haloid, **self._run_kwargs()) - - if os.path.exists(self.halopath): - dataexists = True - else: - dataexists = False + return x, y - - if hasattr(self, 'display_points'): - figure = self.main.display - figure.clear() - ax = figure.add_subplot(111) - - if dataexists: - ax = self.main.display.axes[0] - x,y = self.gethalos_xy() - xmin_use = x.min() - xmax_use = x.max() - ymin_use = y.min() - ymax_use = y.max() - - if 'pos' in self.halo_varx and 'pos' in self.halo_vary and self.fullboxopt == 'specific halo': - self.display_points = ax.plot(x,y,linestyle='-',linewidth=self.main.markersize,color=self.main.markercolor) - else: - self.display_points = ax.plot(x,y,marker=self.main.markerstyle,linestyle='none',markersize=self.main.markersize,color=self.main.markercolor,markeredgecolor=self.main.markercolor) - - ax.set_xlim(xmin_use,xmax_use) - ax.set_ylim(ymin_use,ymax_use) - ax.set_xlabel(self.halo_varx) - ax.set_ylabel(self.halo_vary) - ax.set_xscale(self.x_scale) - ax.set_yscale(self.y_scale) - #self.datastatus = 'X-Y plotted.' - wx.CallAfter(self.main.display.canvas.draw) + + def _plot_button_fired(self): + # The plotting block below used to sit inside the `elif ... == 'zoom'` + # branch, so pressing Plot in parent mode computed a path and then fell + # off the end of the method without drawing anything (AUDIT 2.1.6). + self.halopath = self.catalogue_path() + if not os.path.exists(self.halopath): + self.datastatus = 'Data not found or plotted.' + return + + if hasattr(self, 'display_points'): + figure = self.main.display + figure.clear() + figure.add_subplot(111) + + x, y = self.gethalos_xy() + if len(x) == 0: + return + + ax = self.main.display.axes[0] + if ('pos' in self.halo_varx and 'pos' in self.halo_vary + and self.fullboxopt == 'specific halo'): + self.display_points = ax.plot(x, y, + linestyle='-', + linewidth=self.main.markersize, + color=self.main.markercolor) + else: + self.display_points = ax.plot(x, y, + marker=self.main.markerstyle, + linestyle='none', + markersize=self.main.markersize, + color=self.main.markercolor, + markeredgecolor=self.main.markercolor) + + ax.set_xlim(x.min(), x.max()) + ax.set_ylim(y.min(), y.max()) + ax.set_xlabel(self.halo_varx) + ax.set_ylabel(self.halo_vary) + ax.set_xscale(self.x_scale) + ax.set_yscale(self.y_scale) + wx.CallAfter(self.main.display.canvas.draw) @@ -361,7 +379,7 @@ def _plotxyzrvir_button_changed(self): x = tmphalos['posX'] y = tmphalos['posY'] - z = tmphalos['posY'] + z = tmphalos['posZ'] rvir = tmphalos['rvir'] self.main.scene.mlab.points3d(x, y, z, rvir/1000,colormap="copper") @@ -433,4 +451,6 @@ def __init__(self, main, **kwargs): self.halo_varx = 'posX' self.halo_vary = 'posY' self.gadpath = self.main.headertab.datamasterpath + # Replaces a hardcoded /bigbang absolute path in gethostid(). + self.summarypath = os.path.join(self.gadpath, 'halos') self.halopath = self.main.headertab.parentsimpath + 'RockstarData' diff --git a/src/cme/ui/ics.py b/src/cme/ui/ics.py index cf3c295..2380ba6 100644 --- a/src/cme/ui/ics.py +++ b/src/cme/ui/ics.py @@ -1,20 +1,21 @@ from cme.ui.common import * +# `reWriteIC` post-processes MUSIC's output into the block layout Gadget wants. +# It was previously aliased to `re`, shadowing the standard library module. +import modules.convertfiles.reWriteIC as rewriteic + from cme import cosmology, naming, writers from cme.writers import music #: Side length of the Caterpillar parent box, in Mpc/h. PARENT_BOXLENGTH = 100 -#: Paths to the parent simulation's final snapshot and its initial -#: conditions, relative to the parent simulation directory. Hardcoded to -#: the Caterpillar 512^3 parent; these belong in configuration. +#: Paths to the parent simulation's final snapshot and its initial conditions, +#: relative to the parent simulation directory. Hardcoded to the Caterpillar +#: 512^3 parent; these belong in configuration. PARENT_SNAPSHOT = "/512Parent/outputs/snapdir_063/snap_063" PARENT_ICS = "/512Parent/ics/ics" -from glob import glob -import random -import modules.convertfiles.reWriteIC as re class InitialConditions(HasTraits): @@ -279,15 +280,20 @@ def _generate_button_fired(self): print("EXECUTING...") print(runmusic) subprocess.call(';'.join([cding, runmusic]), shell=True) - re.getBlocks(writepath) + rewriteic.getBlocks(writepath) elif self.parentbox == False: for cosmi in self.cosmologylist: for boxtypei in self.boxtype: for nrviri in self.nrvir: for paddingi in self.padding: - for lmini in self.lmin: - for lmaxi in self.lmax: + # lmin is an Enum holding a single value, not a + # CheckListEditor list like the others. Iterating + # it walked the characters of the string, so a + # levelmin of 10 or above ran twice with the + # wrong values ('1' then '0'). + lmini = self.lmin + for lmaxi in self.lmax: for overlapi in self.overlap: run = dict(region=boxtypei, padding=paddingi, levelmin=lmini, levelmax=lmaxi, @@ -367,7 +373,7 @@ def _generate_button_fired(self): #cpconvert = "cp ./lib/reWriteIC.py ./lib/convertics.py " + writepath #runconvert = "python convertics.py" #rmconvert = "rm reWriteIC.py convertics.py" - re.getBlocks(writepath) + rewriteic.getBlocks(writepath) else: self.confstatus = "Please select only IDs from list." @@ -377,30 +383,47 @@ def _masterpath_changed(self): #self.main.existencetab.masterpath = self.masterpath #self.lagroutputdir = self.main.headertab.masterpath + '/' + self.toplagr - def _haloidselect_changed(self): - self.lagroutputname = 'HALO' + str(self.haloidselect) + 'NRVIR' + str(self.nrvir) - self.resimlagrfile = 'halos' + str(self.toplagr) + '/H' + str(self.haloidselect) + 'NRVIR' + str(self.nrvir) + def _selected_nrvir(self): + """The single chosen nrvir value, or None if not exactly one is selected. + + The trait is a CheckListEditor list, so formatting it with str() yields + "NRVIR['1']" -- which is what the displayed pointer paths used to show + while the files were written with int(self.nrvir[0]) (AUDIT 2.2.4). + """ + return self.nrvir[0] if len(self.nrvir) == 1 else None + + def _refresh_lagr_labels(self): + """Recompute the displayed Lagrangian pointer name and path.""" + nrvir = self._selected_nrvir() + if nrvir is None: + self.lagroutputname = '' + self.resimlagrfile = '' + return + self.lagroutputname = os.path.basename( + naming.lagr_region_path('', self.haloidselect, nrvir)) + self.resimlagrfile = naming.lagr_region_path( + str(self.toplagr), self.haloidselect, nrvir) - self._existencebutton_fired() + def _haloidselect_changed(self): + self._refresh_lagr_labels() + self._existencebutton_fired() def _nrvir_changed(self): - self.lagroutputname = 'HALO' + str(self.haloidselect) + 'NRVIR' + str(self.nrvir) - self.resimlagrfile = str(self.toplagr) + '/H' + str(self.haloidselect) + 'NRVIR' + str(self.nrvir) - self._existencebutton_fired() + self._refresh_lagr_labels() + self._existencebutton_fired() def _toplagr_changed(self): - self.lagroutputdir = str(self.main.headertab.datamasterpath) + str(self.toplagr) - self.lagroutputname = 'H' + str(self.haloidselect) + 'NRVIR' + str(self.nrvir) - self.resimlagrfile = str(self.toplagr) + '/H' + str(self.haloidselect) + 'NRVIR' + str(self.nrvir) - self._existencebutton_fired() + self.lagroutputdir = os.path.join( + str(self.main.headertab.datamasterpath), str(self.toplagr)) + self._refresh_lagr_labels() + self._existencebutton_fired() def _resimlagrdir_changed(self): - self.resimlagrdir = str(self.main.headertab.datamasterpath) - self.resimlagrfile = str(self.toplagr) + '/H' + str(self.haloidselect) + 'NRVIR' + str(self.nrvir) + self.resimlagrdir = str(self.main.headertab.datamasterpath) + self._refresh_lagr_labels() def _resimlagrfile_changed(self): - self.resimlagrdir = str(self.main.headertab.datamasterpath) - self.resimlagrfile = str(self.toplagr) + '/H' + str(self.haloidselect) + 'NRVIR' + str(self.nrvir) + self.resimlagrdir = str(self.main.headertab.datamasterpath) #def _parentsimpath_changed(self): #if len(os.path.basename(glob(self.parentsimpath + "*.conf")[0])) == 1: @@ -571,10 +594,12 @@ def _makeic_button_fired(self): idcand = getcandidatelist(self.candidatefiledir + self.candidatefilename) idcand = idcand[:,0] - for index in range(0,len(idcand)): - if idcand[index] == idhalo: - nhalo = index - + matches = np.flatnonzero(idcand == idhalo) + if len(matches) == 0: + self.filestatus = 'Halo not in candidate list.' + return + nhalo = int(matches[0]) + rvircand = allhalos.loc[idhalo]['rvir'] mvircand = allhalos.loc[idhalo]['mvir'] posXcand = allhalos.loc[idhalo]['posX'] @@ -751,9 +776,7 @@ def __init__(self, main, **kwargs): self.cosmologylist = ['PLANCK'] self.laplaceorder = 6 self.gradorder = 6 - self.lagroutput = str(self.main.headertab.masterpath) + '/' + str(self.toplagr) + '/H' + str(self.haloidselect) + 'NRVIR' + str(self.nrvir) self.resimlagrdir = str(self.main.headertab.datamasterpath) - self.resimlagrfile = '/' + str(self.toplagr) + '/H' + str(self.haloidselect) + 'NRVIR' + str(self.nrvir) self.masterpath = self.main.headertab.masterpath self.outfilename = 'ics' self.parenticpath = self.main.headertab.parentsimpath + 'RockstarData/ics/' @@ -771,7 +794,6 @@ def __init__(self, main, **kwargs): #self.parentsimconf = 'ics_example.conf' self.parentsimpath = self.main.headertab.parentsimpath + '512Parent/ics/' #self.parentsimconf = os.path.basename(glob(self.parentsimpath + "*.conf")[0]) - self.resimdir = str(self.main.headertab.datamasterpath) + str(self.toplagr) + 'H' + str(self.haloidselect) + 'NRVIR' + str(self.nrvir) filename = self.parentsimpath + self.parentsimconf self.parentseedlevel = 9 self.parentseednum = 34567 From 21b918e384c8eefd7ca5a552acafe0fcd9f32d9f Mon Sep 17 00:00:00 2001 From: Brendan Griffen Date: Mon, 27 Jul 2026 22:32:50 +1000 Subject: [PATCH 14/19] refactor: run external commands without a shell Every external invocation was assembled by string concatenation and handed to os.system or subprocess.call(..., shell=True) with paths from Directory traits interpolated in -- 70 call sites. Two consequences that bite in practice on a shared filesystem: - a path containing a space silently splits, so /data/My Runs/H190897 arrives as two arguments; - a path containing ;, $(...) or && executes. All are legal in a directory name. Add cme/shell.py, whose run() takes an argument list and an explicit cwd, and convert the callers: - install.py: 60 os.system calls and 24 os.chdir/chdir-back pairs become a table of source trees and configure flags plus one _build() method. The file drops from 205 lines to 132. The chdir pairs were not exception-safe, so a failed build left the process in the source tree. run_all() also stops at the first failure, where the original `;`-joined strings ran `make install` even after `configure` had failed. Progress now goes to a status field rather than print(), which a GUI user is not necessarily watching. - ics.py: `cd ; ; rm wnoise* temp*` becomes run_music(), which runs MUSIC with cwd set and only cleans up on success. Because the original parts were `;`-joined rather than `&&`-joined, the rm ran even when MUSIC had failed. Scratch removal now globs within one directory instead of shelling out to rm. - gadget.py: the file operations are done in Python. param.txt is written straight into the run directory instead of into the working directory and then `mv`ed; `mkdir -p` becomes os.makedirs(exist_ok=True); and the `tail -n+96 | cat >> | cp` pipeline that splices the generated Config.sh onto the upstream tail becomes install_config_sh(), which needs no temporary file and reports a missing upstream Config.sh instead of silently producing an empty splice. The generated cluster driver script still exists -- it does module loads, ssh and qsub on a remote host, which is genuinely shell work -- but is now invoked as ["bash", path] rather than through a shell, and its name is a constant rather than a literal in two places. No `shell=True` or `os.system` remains in the codebase. --- src/cme/shell.py | 73 +++++++++++++++ src/cme/ui/gadget.py | 82 ++++++++++------- src/cme/ui/ics.py | 39 +++++--- src/cme/ui/install.py | 207 ++++++++++++++---------------------------- 4 files changed, 215 insertions(+), 186 deletions(-) create mode 100644 src/cme/shell.py diff --git a/src/cme/shell.py b/src/cme/shell.py new file mode 100644 index 0000000..2b9e8db --- /dev/null +++ b/src/cme/shell.py @@ -0,0 +1,73 @@ +"""Running external commands without a shell. + +Every external invocation in this codebase used to be assembled by string +concatenation and handed to ``os.system`` or ``subprocess.call(..., shell=True)`` +with GUI-supplied paths interpolated in. That is wrong in two ways that both +bite in practice on a shared cluster: + +* a path containing a space silently splits into two arguments, so + ``/data/My Runs/H190897`` becomes ``/data/My`` and ``Runs/H190897``; +* a path containing shell metacharacters executes. ``;``, ``$(...)`` and ``&&`` + are all legal in a directory name. + +Passing an argument list means the arguments reach the program exactly as +given, with no shell involved. ``cwd`` replaces the ``os.chdir`` / ``chdir`` +back pairs the original used, which leaked the working directory on any +exception. +""" + +import logging +import os +import subprocess + +log = logging.getLogger(__name__) + + +def run(args, cwd=None, check=False): + """Run one command from an argument list, with no shell. + + ``args`` is a sequence whose elements are stringified, so path traits and + numbers can be passed directly. + + Returns the :class:`subprocess.CompletedProcess`. With ``check=True`` a + non-zero exit raises :class:`subprocess.CalledProcessError`. + """ + argv = [str(a) for a in args] + log.info("running %s%s", argv, f" in {cwd}" if cwd else "") + return subprocess.run(argv, cwd=cwd, check=check) + + +def run_all(commands, cwd=None, check=False): + """Run several commands in sequence in the same directory. + + Stops at the first failure and returns the results so far, so a broken + ``configure`` does not lead to ``make install`` running anyway -- which the + original ``;``-joined command strings did. + """ + results = [] + for args in commands: + result = run(args, cwd=cwd, check=check) + results.append(result) + if result.returncode != 0: + log.warning("command failed (%s), stopping: %s", + result.returncode, [str(a) for a in args]) + break + return results + + +def remove_matching(directory, patterns): + """Delete files in ``directory`` matching any of ``patterns``. + + Replaces ``rm wnoise* temp*`` and friends. Globbing in Python keeps the + expansion scoped to one directory: a shell ``rm`` with an empty or + unexpected path variable can match far more than intended. + """ + import glob as globmod + + removed = [] + for pattern in patterns: + for path in globmod.glob(os.path.join(str(directory), pattern)): + if os.path.isfile(path): + os.remove(path) + removed.append(path) + return removed diff --git a/src/cme/ui/gadget.py b/src/cme/ui/gadget.py index 7d77a08..370b34b 100644 --- a/src/cme/ui/gadget.py +++ b/src/cme/ui/gadget.py @@ -1,7 +1,12 @@ from cme.ui.common import * -from cme import cosmology, naming, writers +import shutil + +from cme import cosmology, naming, shell, writers from cme.writers import gadget as gadgetwriter, submit + +#: Driver script the tab writes and then runs. +DRIVER_SCRIPT = "rungadget.sh" import modules.mergertrees.MTCatalogue as MT class GadgetRun(HasTraits): @@ -253,7 +258,7 @@ class GadgetRun(HasTraits): def _subscript_button_fired(self): - f1 = open("rungadget.sh",'w') + f1 = open(DRIVER_SCRIPT, 'w') f1.write("#!/bin/bash \n") if self.subtype == "PBS": @@ -289,20 +294,17 @@ def _subscript_button_fired(self): print("PATH FOUND:",filepath) self.levelmaxuse = int(self.lmax) self.Omega0,self.OmegaLambda,self.OmegaBaryon,self.HubbleParam,sigma8,nspec = cosmology.cosmoconstant(self.cosmologylist[0]) - ext = self.gadpath + 'halos/H' + str(self.haloid) + '/' + foldername + # Written directly into the run directory. The + # original wrote into the current working directory + # and then shelled out to `mv`. + self.makeparam(os.path.join(filepath, 'param.txt'), + filepath, includesub=False) if self.ENABLE_SUBFIND == True: - self.makeparam('param.txt',ext,includesub=False) - self.makeparam('param_sub.txt',ext,includesub=True) - mving1 = "mv param.txt " + filepath - mving2 = "mv param_sub.txt " + filepath - subprocess.call(';'.join([mving1,mving2]), shell=True) - else: - self.makeparam('param.txt',ext,includesub=False) - mving = "mv param.txt " + filepath - subprocess.call([mving], shell=True) - - mkdirs = "mkdir -p " + filepath + "/outputs" - subprocess.call([mkdirs], shell=True) + self.makeparam(os.path.join(filepath, 'param_sub.txt'), + filepath, includesub=True) + + os.makedirs(os.path.join(filepath, "outputs"), + exist_ok=True) scriptpath = os.path.join( filepath, @@ -333,20 +335,12 @@ def _subscript_button_fired(self): writers.write(scriptpath, script) if self.compilegadopt == True: - if self.ENABLE_SUBFIND == True: - self.write_config_sh(os.path.join(filepath, "Config_sub.sh")) - command5 = "tail -n+96 " + gadgetfilepath + "P-Gadget3/Config.sh > " + filepath + "/bottomConfig" - command6 = "cat " + filepath + "/bottomConfig >> " + filepath + "/Config_sub.sh" - command7 = "cp " + filepath + "/Config_sub.sh " + gadgetfilepath + "P-Gadget3/Config.sh" - subprocess.call(';'.join([command5,command6,command7]), shell=True) - else: - self.write_config_sh(os.path.join(filepath, "Config.sh")) - command5 = "tail -n+96 " + gadgetfilepath + "/P-Gadget3/Config.sh > " + filepath + "/bottomConfig" - command6 = "cat " + filepath + "/bottomConfig >> " + filepath + "/Config.sh" - command7 = "cp " + filepath + "/Config.sh " + gadgetfilepath + "P-Gadget3/Config.sh" - subprocess.call(';'.join([command5,command6,command7]), shell=True) - rmconfig = "rm " + filepath + "/bottomConfig" - subprocess.call([rmconfig], shell=True) + name = ("Config_sub.sh" if self.ENABLE_SUBFIND + else "Config.sh") + self.install_config_sh( + os.path.join(filepath, name), + os.path.join(gadgetfilepath, "P-Gadget3", + "Config.sh")) #try: # with open("/bigbang/data/bgriffen/lib/P-Gadget3/P-Gadget3"): pass f1.write("cd " + str(filepath) + "\n") @@ -381,13 +375,33 @@ def _subscript_button_fired(self): f1.close() - command = "bash rungadget.sh" - subprocess.call(';'.join([command]), shell=True) + shell.run(["bash", DRIVER_SCRIPT]) + print("SUBMITTED JOBS!") + + #: Number of leading lines of the upstream Config.sh that this tool replaces. + #: Everything below is the project-specific tail and is preserved verbatim. + CONFIG_PREAMBLE_LINES = 95 - #command = "rm " + str(filepath) + "/rungadget.sh" - #subprocess.call(';'.join([command]), shell=True) + def install_config_sh(self, generated, upstream): + """Splice a generated Config.sh onto the tail of the upstream one. - print("SUBMITTED JOBS!") + The original ran `tail -n+96 > bottomConfig`, `cat bottomConfig + >> `, then `cp ` through a shell, with + both paths interpolated. Doing it in Python removes the injection surface + and the temporary file, and lets a missing upstream file be reported + rather than producing an empty splice. + """ + if not os.path.exists(upstream): + print("Config.sh not found, not splicing:", upstream) + return + + with open(upstream) as f: + tail = f.readlines()[self.CONFIG_PREAMBLE_LINES:] + + with open(generated, "a") as f: + f.writelines(tail) + + shutil.copyfile(generated, upstream) def _refresh_softenings(self): """Mirror the derived softening lengths onto the displayed traits.""" diff --git a/src/cme/ui/ics.py b/src/cme/ui/ics.py index 2380ba6..2da5d15 100644 --- a/src/cme/ui/ics.py +++ b/src/cme/ui/ics.py @@ -4,7 +4,7 @@ # It was previously aliased to `re`, shadowing the standard library module. import modules.convertfiles.reWriteIC as rewriteic -from cme import cosmology, naming, writers +from cme import cosmology, naming, shell, writers from cme.writers import music #: Side length of the Caterpillar parent box, in Mpc/h. @@ -216,6 +216,29 @@ class InitialConditions(HasTraits): Item(name='confstatus',label='Status',style='readonly'), Group(Item(name='executemusic',label='Execute?'),Item(name='generate_button',show_label=False),enabled_when='haloidselect in haloid'),label='Output'))) + #: Scratch files MUSIC leaves in the output directory. + MUSIC_SCRATCH = ("wnoise*", "temp*") + + def run_music(self, confname, workdir, cleanup=False): + """Invoke MUSIC on a configuration file, optionally tidying up after. + + Previously this joined `cd `, the MUSIC command and `rm wnoise* + temp*` with semicolons and ran them through a shell. Both paths come + from Directory traits, so a space split the command and a metacharacter + executed; and because the parts were `;`-joined rather than `&&`-joined, + the `rm` ran even when MUSIC had failed. + """ + binary = os.path.join(str(self.musicpath), "MUSIC") + self.confstatus = f"Running MUSIC on {os.path.basename(confname)}..." + result = shell.run([binary, confname], cwd=workdir) + if result.returncode != 0: + self.confstatus = f"MUSIC failed ({result.returncode})." + return result + if cleanup: + shell.remove_matching(workdir, self.MUSIC_SCRATCH) + self.confstatus = "Generated initial conditions." + return result + def _parentbox_changed(self): if self.parentbox == True: self.resimbox = False @@ -275,11 +298,7 @@ def _generate_button_fired(self): )) self.confstatus = 'Generated halo configuration files.' - runmusic = self.musicpath + '/MUSIC ' + confname - cding = "cd " + writepath - print("EXECUTING...") - print(runmusic) - subprocess.call(';'.join([cding, runmusic]), shell=True) + self.run_music(confname, writepath) rewriteic.getBlocks(writepath) elif self.parentbox == False: @@ -364,12 +383,8 @@ def _generate_button_fired(self): self.confstatus = 'Generated halo configuration files.' if self.executemusic == True: - runmusic = self.musicpath + '/MUSIC ' + confname - cding = "cd " + writepath - rmfiles = "rm wnoise* temp*" - print("EXECUTING...") - print(runmusic) - subprocess.call(';'.join([cding,runmusic,rmfiles]), shell=True) + self.run_music(confname, writepath, + cleanup=True) #cpconvert = "cp ./lib/reWriteIC.py ./lib/convertics.py " + writepath #runconvert = "python convertics.py" #rmconvert = "rm reWriteIC.py convertics.py" diff --git a/src/cme/ui/install.py b/src/cme/ui/install.py index 6296e0d..1af7549 100644 --- a/src/cme/ui/install.py +++ b/src/cme/ui/install.py @@ -1,8 +1,12 @@ from cme.ui.common import * +from cme import shell + class InstallFiles(HasTraits): - view = View(VGroup( HGroup(Item('libpath',show_label=False,springy=True), + view = View(VGroup(HGroup(Item('status', style='readonly', + show_label=False, springy=True)), + HGroup(Item('libpath',show_label=False,springy=True), Item('makelibdir_button',show_label=False,springy=True)), HGroup(Item('fftw2dir',show_label=False,springy=True), Item('installfftw2_button',show_label=False,springy=True)), @@ -18,6 +22,7 @@ class InstallFiles(HasTraits): Item('installszip_button',show_label=False,springy=True)))) libpath = Directory + status = Str fftw2dir = Directory fftw3dir = Directory gsldir = Directory @@ -41,157 +46,79 @@ def _libpath_changed(self): self.zlibdir = self.libpath + '/zlib' self.szipdir = self.libpath + '/szip' + #: Source trees the install buttons build, and the configure flag sets for + #: each. FFTW ships four variants (single/double precision, with and without + #: the type prefix), so all four are built in turn. + SOURCES = { + "fftw2": ("./lib/installs/fftw-2.1.5/", [ + ["--enable-type-prefix", "--enable-mpi"], + ["--enable-float", "--enable-type-prefix", "--enable-mpi"], + ["--enable-mpi"], + ["--enable-float", "--enable-mpi"], + ]), + "fftw3": ("./lib/installs/fftw-3.3.3/", [ + ["--enable-threads", "--enable-float"], + ["--enable-threads"], + ["--enable-type-prefix", "--enable-threads", "--enable-float"], + ["--enable-type-prefix", "--enable-threads"], + ]), + "hdf5": ("./lib/installs/hdf5-1.8.10/", [ + ["--enable-fortran", "--enable-cxx"], + ]), + "gsl": ("./lib/installs/gsl-1.9/", [[]]), + "szip": ("./lib/installs/szip-2.1/", [ + ["--enable-shared=no", "--enable-static=yes"], + ]), + "zlib": ("./lib/installs/zlib-1.2.8/", [[]]), + } + + def _build(self, name, prefix): + """Configure, build and install one dependency into ``prefix``. + + Each variant runs as argument lists with ``cwd`` set, rather than + ``os.chdir`` plus ``os.system`` on an interpolated string: ``prefix`` + comes from a Directory trait, so a path containing a space previously + split the command and one containing a shell metacharacter executed. + ``run_all`` also stops at the first failure, where the original + ``;``-joined strings ran ``make install`` even after ``configure`` failed. + """ + source, variants = self.SOURCES[name] + if not os.path.isdir(source): + self.status = f"Source tree not found: {source}" + return + + prefix = str(prefix) + self.status = f"Installing {name} into {prefix}..." + for flags in variants: + shell.run_all([ + ["make", "clean"], + ["./configure", f"--prefix={prefix}", *flags], + ["make"], + ["make", "install"], + ["make", "clean"], + ], cwd=source) + self.status = f"{name.upper()} installed: {prefix}" + def _makelibdir_button_fired(self): - if not os.path.exists(str(self.libpath)): - os.makedirs(str(self.libpath)) + os.makedirs(str(self.libpath), exist_ok=True) def _installfftw2_button_fired(self): - tmplibpath = './lib/installs/fftw-2.1.5/' - print("Installing:",tmplibpath) - - flagstr = '--enable-type-prefix --enable-mpi' - tmpdir = os.getcwd() - os.chdir(tmplibpath) - os.system("make clean") - os.system("./configure --prefix=%s %s" % (str(self.fftw2dir),flagstr)) - os.system("make") - os.system("make install") - os.system("make clean") - os.chdir(tmpdir) - - flagstr = '--enable-float --enable-type-prefix --enable-mpi' - tmpdir = os.getcwd() - os.chdir(tmplibpath) - os.system("make clean") - os.system("./configure --prefix=%s %s" % (str(self.fftw2dir),flagstr)) - os.system("make") - os.system("make install") - os.system("make clean") - os.chdir(tmpdir) - - flagstr = '--enable-mpi' - tmpdir = os.getcwd() - os.chdir(tmplibpath) - os.system("make clean") - os.system("./configure --prefix=%s %s" % (str(self.fftw2dir),flagstr)) - os.system("make") - os.system("make install") - os.system("make clean") - os.chdir(tmpdir) - - flagstr = '--enable-float --enable-mpi' - tmpdir = os.getcwd() - os.chdir(tmplibpath) - os.system("make clean") - os.system("./configure --prefix=%s %s" % (str(self.fftw2dir),flagstr)) - os.system("make") - os.system("make install") - os.system("make clean") - os.chdir(tmpdir) - - print("FFTW2 INSTALLED:",str(self.fftw2dir)) + self._build("fftw2", self.fftw2dir) def _installfftw3_button_fired(self): - tmplibpath = './lib/installs/fftw-3.3.3/' - print("Installing:",tmplibpath) - - flagstr = '--enable-threads --enable-float' - tmpdir = os.getcwd() - os.chdir(tmplibpath) - os.system("make clean") - os.system("./configure --prefix=%s %s" % (str(self.fftw3dir),flagstr)) - os.system("make") - os.system("make install") - os.system("make clean") - os.chdir(tmpdir) - - flagstr = '--enable-threads' - tmpdir = os.getcwd() - os.chdir(tmplibpath) - os.system("make clean") - os.system("./configure --prefix=%s %s" % (str(self.fftw3dir),flagstr)) - os.system("make") - os.system("make install") - os.system("make clean") - os.chdir(tmpdir) - - flagstr = '--enable-type-prefix --enable-threads --enable-float ' - tmpdir = os.getcwd() - os.chdir(tmplibpath) - os.system("make clean") - os.system("./configure --prefix=%s %s" % (str(self.fftw3dir),flagstr)) - os.system("make") - os.system("make install") - os.system("make clean") - os.chdir(tmpdir) - - flagstr = '--enable-type-prefix --enable-threads ' - tmpdir = os.getcwd() - os.chdir(tmplibpath) - os.system("make clean") - os.system("./configure --prefix=%s %s" % (str(self.fftw3dir),flagstr)) - os.system("make") - os.system("make install") - os.system("make clean") - os.chdir(tmpdir) - - print("FFTW3 INSTALLED:",str(self.fftw3dir)) + self._build("fftw3", self.fftw3dir) def _installhdf5_button_fired(self): - tmplibpath = './lib/installs/hdf5-1.8.10/' - print("Installing:",tmplibpath) - #(/home/bgriffen/lib/szip,/home/bgriffen/lib/zlib/include,/home/bgriffen/lib/zlib/lib) - #flagstr = '--enable-fortran --enable-cxx --with-szlib=%s --with-zlib=%s,%s' - flagstr = '--enable-fortran --enable-cxx' - tmpdir = os.getcwd() - os.chdir(tmplibpath) - os.system("make clean") - os.system("./configure --prefix=%s %s" % (str(self.hdf5dir),flagstr)) - os.system("make") - os.system("make install") - os.system("make clean") - os.chdir(tmpdir) - print("HDF5 INSTALLED:",str(self.hdf5dir)) - + self._build("hdf5", self.hdf5dir) + def _installgsl_button_fired(self): - tmplibpath = './lib/installs/gsl-1.9/' - print("Installing:",tmplibpath) - tmpdir = os.getcwd() - os.chdir(tmplibpath) - os.system("make clean") - os.system("./configure --prefix=%s" % str(self.gsldir)) - os.system("make") - os.system("make install") - os.system("make clean") - os.chdir(tmpdir) - print("GSL INSTALLED:",str(self.gsldir)) + self._build("gsl", self.gsldir) def _installszip_button_fired(self): - tmplibpath = './lib/installs/szip-2.1/' - print("Installing:",tmplibpath) - tmpdir = os.getcwd() - os.chdir(tmplibpath) - os.system("make clean") - flagstr = '--enable-shared=no --enable-static=yes' - os.system("./configure --prefix=%s %s" % (str(self.szipdir),flagstr)) - os.system("make") - os.system("make install") - os.system("make clean") - os.chdir(tmpdir) - print("SZIP INSTALLED:",str(self.szipdir)) + self._build("szip", self.szipdir) def _installzlib_button_fired(self): - tmplibpath = './lib/installs/zlib-1.2.8/' - print("Installing:",tmplibpath) - tmpdir = os.getcwd() - os.chdir(tmplibpath) - os.system("make clean") - os.system("./configure --prefix=%s" % str(self.zlibdir)) - os.system("make") - os.system("make install") - os.system("make clean") - os.chdir(tmpdir) - print("ZLIB INSTALLED:",str(self.zlibdir)) + self._build("zlib", self.zlibdir) def __init__(self, main, **kwargs): HasTraits.__init__(self) From 4e182b219e25a93f44fbffcc446713757174d7f4 Mon Sep 17 00:00:00 2001 From: Brendan Griffen Date: Mon, 27 Jul 2026 22:37:23 +1000 Subject: [PATCH 15/19] feat: load cluster profiles from clusters.toml The Home tab identified the machine with an if-chain on platform.node() and then assigned hardcoded absolute paths, so the tool only worked for its original author on five named machines. Anyone else had to edit the source. It carried 19 personal identifiers between home.py and gadget.py: usernames (bgriffen, uqbgriff), hostnames (csr-dyn-150.mit.edu, bigbang.mit.edu, rclogin13.rc.fas.harvard.edu, Brendans-MacBook-Pro.local), absolute paths (/bigbang/data, /n/home01/bgriffen/data, /Users/griffen/Desktop/cme) and two email addresses. clusters.toml describes each machine as data: which hostnames select it, where home directories live, and where MUSIC, P-Gadget3 and the project data sit, using {home}/{user}/{master} placeholders. cme/config.py loads it from $CME_CLUSTERS, then ~/.config/cme/clusters.toml, then the repository copy, and falls back to a single local profile so the application still starts on a machine with no configuration and no TOML parser. The two path derivations are now one. Previously _clusteropt_changed and _username_changed computed *different* path sets from the same inputs -- the latter derived parentsimpath as {master}/AnnaGroup/caterpillar/parent/512Parent/ while the hostname branches used {home}/AnnaGroup/caterpillar/parent -- so which paths you ended up with depended on the order you touched the fields. Both handlers now call one _apply_profile(). The Gadget tab picks up its scheduler, queue and extra directives from the profile, and takes the Gadget source tree from the Home tab instead of two hardcoded absolute paths selected by scheduler type. Its `ssh antares` hop and the `unloadmods`/`loadgadget` lines become configurable rather than baked in; the matching `logout` that closed the ssh heredoc goes with it. SLURMqueue was an Enum of three Harvard queue names, which could not hold the queue another site's profile specifies, and is now a Str. 24 tests, including that every hostname the old if-chain handled still resolves to the right profile, that no placeholder survives resolution, and that no resolved path contains a personal home directory. --- clusters.toml | 78 +++++++++++++++++++++++ src/cme/config.py | 134 ++++++++++++++++++++++++++++++++++++++++ src/cme/ui/gadget.py | 49 +++++++++------ src/cme/ui/home.py | 143 ++++++++++++++++++------------------------- tests/test_config.py | 128 ++++++++++++++++++++++++++++++++++++++ 5 files changed, 429 insertions(+), 103 deletions(-) create mode 100644 clusters.toml create mode 100644 src/cme/config.py create mode 100644 tests/test_config.py diff --git a/clusters.toml b/clusters.toml new file mode 100644 index 0000000..6eb652c --- /dev/null +++ b/clusters.toml @@ -0,0 +1,78 @@ +# Cluster profiles. +# +# Each [clusters.] section describes one machine: where home directories +# live, and where MUSIC, P-Gadget3 and the project data sit relative to it. +# These were previously an if-chain on platform.node() inside the Home tab, with +# absolute paths, usernames and email addresses baked into the source, so nobody +# but the original author could use the tool without editing it. +# +# Placeholders expanded in every path value: +# {home} the profile's home_root, or the current user's home directory +# {user} the profile's `username`, or the login name +# {master} {home}/{user} +# +# `hostnames` lists the platform.node() values that select this profile +# automatically. A profile with no hostnames must be selected by hand. +# Copy this file to ~/.config/cme/clusters.toml to override it. + +default = "local" + +[clusters.local] +# Fallback when the hostname matches nothing. Everything under the user's home. +hostnames = [] +gadget = "{master}/lib/P-Gadget3" +music = "{master}/lib/music" +data = "{master}/projects/caterpillar/data" +parent = "{master}/projects/caterpillar/parent" + +[clusters.bigbang] +hostnames = ["bigbang.mit.edu"] +home_root = "/bigbang/data" +gadget = "{master}/lib/P-Gadget3" +music = "{master}/lib/music" +data = "{home}/AnnaGroup/caterpillar" +parent = "{home}/AnnaGroup/caterpillar/parent" +scheduler = "PBS" +queue = "default" +# Torque needs its OpenMPI environment sourced before mpirun. +extra_lines = "source /opt/torque/etc/openmpi-setup.sh" +extra_flags = "#PBS -m ae" + +[clusters.odyssey] +hostnames = ["rclogin13.rc.fas.harvard.edu"] +home_root = "/n/home01" +gadget = "{master}/lib/P-Gadget3" +music = "{master}/lib/music" +data = "{home}/caterpillar" +parent = "{home}/caterpillar/parent" +scheduler = "SLURM" +queue = "itc_cluster" + +[clusters.spacebase] +hostnames = ["spacebase"] +home_root = "/spacebase/data" +gadget = "{master}/lib/P-Gadget3" +music = "{master}/lib/music" +data = "{home}/AnnaGroup/caterpillar" +parent = "{home}/AnnaGroup/caterpillar/parent" +scheduler = "PBS" +queue = "default" + +[clusters.antares] +hostnames = ["antares"] +gadget = "{master}/lib/P-Gadget3" +music = "{master}/lib/music" +data = "{master}/projects/caterpillar/data" +parent = "{master}/projects/caterpillar/parent" +scheduler = "PBS" +queue = "default" + +[clusters.barrine] +hostnames = ["barrine"] +username = "uqbgriff" +gadget = "{master}/lib/P-Gadget3" +music = "{master}/lib/music" +data = "{master}/projects/caterpillar/data" +parent = "{master}/projects/caterpillar/parent" +scheduler = "PBS" +queue = "default" diff --git a/src/cme/config.py b/src/cme/config.py new file mode 100644 index 0000000..ec51330 --- /dev/null +++ b/src/cme/config.py @@ -0,0 +1,134 @@ +"""Cluster profiles loaded from ``clusters.toml``. + +Replaces the ``platform.node()`` if-chain in the Home tab, which hardcoded +absolute paths, two usernames and an email address, so the tool only worked for +its original author on five named machines. + +Lookup order for the configuration file: + +1. ``$CME_CLUSTERS`` if set +2. ``~/.config/cme/clusters.toml`` +3. ``clusters.toml`` beside the installed package, or at the repository root +""" + +import getpass +import os +import platform + +try: # Python 3.11+ + import tomllib +except ModuleNotFoundError: # pragma: no cover - 3.9/3.10 fallback + try: + import tomli as tomllib + except ModuleNotFoundError: + tomllib = None + +#: Keys whose values get placeholder expansion. +PATH_KEYS = ("gadget", "music", "data", "parent") + +_FALLBACK = { + "default": "local", + "clusters": { + "local": { + "hostnames": [], + "gadget": "{master}/lib/P-Gadget3", + "music": "{master}/lib/music", + "data": "{master}/projects/caterpillar/data", + "parent": "{master}/projects/caterpillar/parent", + } + }, +} + + +def config_path(): + """Path to the clusters file that will be used, or None if none is found.""" + override = os.environ.get("CME_CLUSTERS") + if override: + return override if os.path.exists(override) else None + + candidates = [ + os.path.expanduser("~/.config/cme/clusters.toml"), + os.path.join(os.path.dirname(os.path.dirname(os.path.dirname( + os.path.abspath(__file__)))), "clusters.toml"), + os.path.join(os.path.dirname(os.path.abspath(__file__)), "clusters.toml"), + ] + for path in candidates: + if os.path.exists(path): + return path + return None + + +def load(path=None): + """Read the clusters file, falling back to a single local profile. + + The fallback keeps the application usable on a machine with no + configuration and no TOML parser available, rather than failing to start. + """ + path = path or config_path() + if path is None or tomllib is None: + return _FALLBACK + with open(path, "rb") as handle: + data = tomllib.load(handle) + if not data.get("clusters"): + return _FALLBACK + return data + + +def profile_names(config=None): + """Available profile names, for the Home tab's cluster selector.""" + return sorted((config or load())["clusters"]) + + +def detect(config=None, hostname=None): + """Name of the profile matching this machine, or the configured default.""" + config = config or load() + hostname = hostname or platform.node() + for name, profile in config["clusters"].items(): + if hostname in profile.get("hostnames", []): + return name + default = config.get("default") + if default in config["clusters"]: + return default + return profile_names(config)[0] + + +def resolve(name=None, config=None, username=None): + """Return one profile's paths with placeholders expanded. + + >>> paths = resolve("local", username="alice") + >>> paths["master"].endswith("alice") + True + + Keys: ``name``, ``username``, ``home``, ``master``, plus the entries in + :data:`PATH_KEYS` and any scheduler settings the profile carries. + """ + config = config or load() + name = name or detect(config) + if name not in config["clusters"]: + raise ValueError( + f"unknown cluster {name!r}; expected one of {profile_names(config)}" + ) + profile = config["clusters"][name] + + user = username or profile.get("username") or getpass.getuser() + home = profile.get("home_root") or os.path.expanduser("~") + # A home_root is the directory *containing* per-user homes, so the user's + # own directory sits below it. Without a home_root, the expanded home + # directory already includes the user name. + master = os.path.join(home, user) if profile.get("home_root") else home + + fields = {"home": home, "user": user, "master": master} + resolved = { + "name": name, + "username": user, + "home": home, + "master": master, + "scheduler": profile.get("scheduler", "PBS"), + "queue": profile.get("queue", "default"), + "extra_lines": profile.get("extra_lines", ""), + "extra_flags": profile.get("extra_flags", ""), + } + for key in PATH_KEYS: + template = profile.get(key) + resolved[key] = template.format(**fields) if template else "" + return resolved diff --git a/src/cme/ui/gadget.py b/src/cme/ui/gadget.py index 370b34b..014c73c 100644 --- a/src/cme/ui/gadget.py +++ b/src/cme/ui/gadget.py @@ -14,14 +14,13 @@ class GadgetRun(HasTraits): PBSncores = Range(0,64,8) PBSnnodes = Range(0,10,8) - username = Str("bgriffen") - PBSemail = Str("bgriffen@space.mit.edu") + username = Str() PBSjobname = Str() PBSqueue = Str("default") outfile = Str("OUTPUT") errorfile = Str("ERROR") PBSextraflags = Str("#PBS -m ae ") - PBSextralines = Str("source /opt/torque/etc/openmpi-setup.sh") + PBSextralines = Str() PBSexecute = Str() compilegadopt = Bool(False) InitCondFile = Str() @@ -127,11 +126,17 @@ class GadgetRun(HasTraits): SLURMcores = Range(64,512,64) SLURMtime = Int(5000) SLURMmemory = Int(3500) - SLURMqueue = Enum(['itc_cluster','general','serial_requeue']) - emailaddress = Str('brendan.f.griffen@gmail.com') + # Was an Enum of three Harvard queue names, which cannot hold the queue a + # different site's profile specifies. + SLURMqueue = Str("general") + emailaddress = Str() subtype = Enum(['SLURM','PBS']) SLURMexecute = Str() + #: Environment setup run before building Gadget on a SLURM machine. + #: Previously the hardcoded lines `unloadmods` and `loadgadget`. + modulelines = Str("unloadmods\nloadgadget") + view = View(Group(Group(Group( Item('gadpath',label='Base Path'), Item(name='haloid',label='Halo ID'), @@ -261,18 +266,18 @@ def _subscript_button_fired(self): f1 = open(DRIVER_SCRIPT, 'w') f1.write("#!/bin/bash \n") - if self.subtype == "PBS": - gadgetfilepath = "/bigbang/data/bgriffen/lib/" + # The Gadget source tree comes from the Home tab, which derives it from the + # selected cluster profile. This used to be two hardcoded absolute paths + # under a specific user's home directory, chosen by scheduler type. + gadgetpath = str(self.main.headertab.gadgetpath) + gadgetfilepath = os.path.dirname(gadgetpath.rstrip(os.sep)) - f1.write("cd " + gadgetfilepath + "P-Gadget3\n") - f1.write("ssh antares <<'ENDSSH' \n") - if self.subtype == "SLURM": - gadgetfilepath = "/n/home01/bgriffen/data/lib/" - f1.write("unloadmods\n") - f1.write("loadgadget\n") - f1.write("cd " + gadgetfilepath + "P-Gadget3\n") - + for line in self.modulelines.splitlines(): + f1.write(line + "\n") + + f1.write("cd " + gadgetpath + "\n") + f1.write("make clean\n") f1.write("make -j 8\n") @@ -370,9 +375,6 @@ def _subscript_button_fired(self): elif self.subtype == "SLURM": f1.write("sbatch runscript \n") - if self.subtype == "PBS": - f1.write("logout \n") - f1.close() shell.run(["bash", DRIVER_SCRIPT]) @@ -726,6 +728,17 @@ def __init__(self, main, **kwargs): self.FOF_SECONDARY_LINK_TYPES = '4+8+16+32' self.ExpansionListArr = np.zeros(shape=(1,4)) self.clusteropt = self.main.headertab.clusteropt + # Scheduler, queue and any extra directives come from the cluster + # profile in clusters.toml rather than from hardcoded defaults. + profile = self.main.headertab.scheduler_defaults() + self.subtype = profile['scheduler'] + self.username = profile['username'] + if profile['scheduler'] == 'SLURM': + self.SLURMqueue = profile['queue'] + else: + self.PBSqueue = profile['queue'] + self.PBSextralines = profile['extra_lines'] + self.PBSextraflags = profile['extra_flags'] self._refresh_execute() #self.PBSstring = \ diff --git a/src/cme/ui/home.py b/src/cme/ui/home.py index 0abf7b6..205c0a9 100644 --- a/src/cme/ui/home.py +++ b/src/cme/ui/home.py @@ -1,18 +1,25 @@ +"""Home tab: pick a cluster profile and confirm the derived paths.""" + from cme.ui.common import * -import getpass -from os.path import expanduser + +from cme import config + + class Header(HasTraits): - view = View(Item(name='clusteropt',label='Cluster' ,padding=5), - Item(name='currentdir' ,label='CWD',style='readonly',padding=5), - Item(name='username' ,label='Username',padding=5), - Item(name='masterpath',label='Home',padding=5), - Item(name='gadgetpath',label='Gadget',padding=5), - Item(name='musicpath',label='Music',padding=5), - Item(name='parentsimpath',label='Parent Sim.',padding=5), - Item(name='datamasterpath',label='Project Data',padding=5)) - - clusteropt = Enum(['antares','barrine','odyssey','macbook','spacebase','bigbang','macbook']) + view = View(Item(name='clusteropt', label='Cluster', padding=5), + Item(name='configfile', label='Profiles', style='readonly', padding=5), + Item(name='currentdir', label='CWD', style='readonly', padding=5), + Item(name='username', label='Username', padding=5), + Item(name='masterpath', label='Home', padding=5), + Item(name='gadgetpath', label='Gadget', padding=5), + Item(name='musicpath', label='Music', padding=5), + Item(name='parentsimpath', label='Parent Sim.', padding=5), + Item(name='datamasterpath', label='Project Data', padding=5)) + + clusteropt = Enum(values='clusternames') + clusternames = List(Str) + configfile = Str username = Str masterpath = Directory homepath = Directory @@ -22,85 +29,51 @@ class Header(HasTraits): parentsimpath = Directory currentdir = Directory - def _username_changed(self): - self.masterpath = self.homepath + self.username - self.gadgetpath = self.masterpath + '/lib/P-Gadget3/' - self.parentsimpath = self.masterpath + '/AnnaGroup/caterpillar/parent/512Parent/' - self.datamasterpath = self.masterpath + '/projects/caterpillar/data/' - self.musicpath = self.homepath + self.username + '/lib/music/' - - def _clusteropt_changed(self): - self.homepath = expanduser("~") + def _apply_profile(self): + """Recompute every derived path from the selected profile and username. -# if self.clusteropt == 'Macbook': -# self.homepath = '/home/' -# -# if self.clusteropt == 'antares': -# self.homepath = '/home/' -# -# if self.clusteropt == 'barrine': -# self.homepath = '/home/' -# -# if self.clusteropt == 'odyssey': -# self.homepath = '/n/home01/' -# -# if self.clusteropt == 'spacebase': -# self.homepath = '/spacebase/data/' -# - self.masterpath = self.homepath + self.username + This replaces an if-chain on platform.node() that hardcoded absolute + paths and two usernames, plus a _username_changed handler that derived a + different and inconsistent set of paths from the same inputs. + """ + paths = config.resolve(self.clusteropt, + config=self._config, + username=self.username or None) + self.homepath = paths['home'] + self.masterpath = paths['master'] + self.gadgetpath = paths['gadget'] + self.musicpath = paths['music'] + self.datamasterpath = paths['data'] + self.parentsimpath = paths['parent'] - def __init__(self, main, **kwargs): - self.username = getpass.getuser() - self.homepath = expanduser("~") - if platform.node() == "csr-dyn-150.mit.edu": - self.clusteropt = 'macbook' - self.homepath = '/Users/griffen/Desktop/cme/' - self.masterpath = self.homepath + self.username - self.gadgetpath = self.masterpath + '/lib/P-Gadget3' - self.musicpath = self.masterpath+ '/lib/music' - self.datamasterpath = self.homepath + 'AnnaGroup/caterpillar/' - self.parentsimpath = self.homepath + 'AnnaGroup/caterpillar/parent/' - #self.homepath = '/Users/' -# - if platform.node() == "Brendans-MacBook-Pro.local": - self.homepath = '/Users/' -# - if platform.node() == 'antares': - self.clusteropt = 'antares' + def _username_changed(self): + self._apply_profile() - if platform.node() == 'barrine': - self.username = 'uqbgriff' + def _clusteropt_changed(self): + self._apply_profile() - if platform.node() == 'rclogin13.rc.fas.harvard.edu': - self.clusteropt = 'odyssey' - self.homepath = '/n/home01/bgriffen/data/' - self.masterpath = self.homepath + self.username - self.gadgetpath = self.masterpath + '/lib/P-Gadget3' - self.musicpath = self.masterpath + '/lib/music' - self.datamasterpath = self.homepath + 'caterpillar/' - self.parentsimpath = self.homepath + 'caterpillar/parent/' + def scheduler_defaults(self): + """Scheduler, queue and extra directives for the selected profile. - if platform.node() == 'spacebase': - self.clusteropt = 'spacebase' - self.homepath = '/spacebase/data/' - self.masterpath = self.homepath + self.username - self.gadgetpath = self.masterpath + '/lib/P-Gadget3' - self.musicpath = self.masterpath+ '/lib/music' - self.datamasterpath = self.homepath + 'AnnaGroup/caterpillar/' - self.parentsimpath = self.homepath + 'AnnaGroup/caterpillar/parent/' + The Gadget tab reads these instead of carrying hardcoded queue names. + """ + return config.resolve(self.clusteropt, + config=self._config, + username=self.username or None) - if platform.node() == 'bigbang.mit.edu': - self.clusteropt = 'bigbang' - self.homepath = '/bigbang/data/' - self.masterpath = self.homepath + self.username - self.gadgetpath = self.masterpath + '/lib/P-Gadget3' - self.musicpath = self.masterpath + '/lib/music' - self.datamasterpath = self.homepath + 'AnnaGroup/caterpillar/' - self.parentsimpath = self.homepath + 'AnnaGroup/caterpillar/parent/' + def __init__(self, main, **kwargs): + HasTraits.__init__(self) + self.main = main + self._config = config.load() + self.clusternames = config.profile_names(self._config) + self.configfile = config.config_path() or '(built-in default)' self.currentdir = os.getcwd() - self.masterpath = self.homepath + self.username - - HasTraits.__init__(self) - self.main = main + detected = config.detect(self._config) + paths = config.resolve(detected, config=self._config) + self.username = paths['username'] + # Assigning clusteropt fires _clusteropt_changed, which applies the + # profile; set it last so the username is already in place. + self.clusteropt = detected + self._apply_profile() diff --git a/tests/test_config.py b/tests/test_config.py new file mode 100644 index 0000000..2ea17c9 --- /dev/null +++ b/tests/test_config.py @@ -0,0 +1,128 @@ +import os + +import pytest + +from cme import config + +REPO_CLUSTERS = os.path.join( + os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "clusters.toml" +) + +pytestmark = pytest.mark.skipif( + config.tomllib is None, reason="no TOML parser available" +) + + +@pytest.fixture +def cfg(): + return config.load(REPO_CLUSTERS) + + +class TestLoad: + def test_reads_the_repository_profiles(self, cfg): + assert set(cfg["clusters"]) >= { + "local", "bigbang", "odyssey", "spacebase", "antares", "barrine" + } + + def test_falls_back_to_a_local_profile_when_no_file_is_found(self, monkeypatch): + # With no config file and no override, the application must still start. + monkeypatch.setattr(config, "config_path", lambda: None) + fallback = config.load() + assert list(fallback["clusters"]) == ["local"] + assert fallback["default"] == "local" + + def test_env_override_selects_the_file(self, monkeypatch): + monkeypatch.setenv("CME_CLUSTERS", REPO_CLUSTERS) + assert config.config_path() == REPO_CLUSTERS + + def test_env_override_pointing_nowhere_is_ignored(self, monkeypatch): + monkeypatch.setenv("CME_CLUSTERS", "/nonexistent/clusters.toml") + assert config.config_path() is None + + def test_profile_names_are_sorted(self, cfg): + names = config.profile_names(cfg) + assert names == sorted(names) + + +class TestDetect: + @pytest.mark.parametrize( + "hostname,expected", + [ + ("bigbang.mit.edu", "bigbang"), + ("rclogin13.rc.fas.harvard.edu", "odyssey"), + ("spacebase", "spacebase"), + ("antares", "antares"), + ("barrine", "barrine"), + ], + ) + def test_recognises_the_hostnames_the_if_chain_handled( + self, cfg, hostname, expected + ): + assert config.detect(cfg, hostname=hostname) == expected + + def test_unknown_hostname_falls_back_to_the_default(self, cfg): + assert config.detect(cfg, hostname="some-laptop.local") == "local" + + +class TestResolve: + def test_placeholders_are_expanded(self, cfg): + paths = config.resolve("bigbang", config=cfg, username="alice") + assert paths["master"] == "/bigbang/data/alice" + assert paths["gadget"] == "/bigbang/data/alice/lib/P-Gadget3" + assert paths["music"] == "/bigbang/data/alice/lib/music" + assert paths["data"] == "/bigbang/data/AnnaGroup/caterpillar" + + def test_no_placeholder_survives_resolution(self, cfg): + for name in config.profile_names(cfg): + paths = config.resolve(name, config=cfg, username="alice") + for key in config.PATH_KEYS: + assert "{" not in paths[key], (name, key) + + def test_home_root_puts_the_user_below_it(self, cfg): + # /n/home01 contains per-user homes, so alice's is /n/home01/alice. + paths = config.resolve("odyssey", config=cfg, username="alice") + assert paths["master"] == "/n/home01/alice" + + def test_local_profile_uses_the_real_home_directory(self, cfg): + paths = config.resolve("local", config=cfg, username="alice") + assert paths["master"] == os.path.expanduser("~") + + def test_profile_username_is_used_when_none_is_given(self, cfg): + # The barrine profile carries a site-specific login name, which used to + # be assigned from an if-chain in the Home tab. + assert config.resolve("barrine", config=cfg)["username"] == "uqbgriff" + + def test_explicit_username_overrides_the_profile(self, cfg): + paths = config.resolve("barrine", config=cfg, username="alice") + assert paths["username"] == "alice" + + def test_scheduler_settings_are_exposed(self, cfg): + assert config.resolve("odyssey", config=cfg)["scheduler"] == "SLURM" + assert config.resolve("odyssey", config=cfg)["queue"] == "itc_cluster" + assert config.resolve("bigbang", config=cfg)["scheduler"] == "PBS" + + def test_torque_profile_carries_its_mpi_setup_line(self, cfg): + extra = config.resolve("bigbang", config=cfg)["extra_lines"] + assert "openmpi-setup.sh" in extra + + def test_scheduler_defaults_to_pbs(self, cfg): + assert config.resolve("local", config=cfg)["scheduler"] == "PBS" + + def test_rejects_an_unknown_profile(self, cfg): + with pytest.raises(ValueError, match="unknown cluster"): + config.resolve("nonexistent", config=cfg) + + +class TestNoPersonalDataInProfiles: + """The if-chain this replaced hardcoded one author's paths and addresses.""" + + def test_profiles_contain_no_email_addresses(self): + with open(REPO_CLUSTERS) as handle: + assert "@" not in handle.read() + + def test_no_profile_hardcodes_a_personal_home_directory(self, cfg): + for name in config.profile_names(cfg): + paths = config.resolve(name, config=cfg, username="alice") + for key in config.PATH_KEYS: + assert "bgriffen" not in paths[key], (name, key) + assert "griffen" not in paths[key], (name, key) From 6d8c6ab15b858250ad5e52adf16928bb3751e623 Mon Sep 17 00:00:00 2001 From: Brendan Griffen Date: Mon, 27 Jul 2026 22:54:42 +1000 Subject: [PATCH 16/19] chore: remove dead code, close file handles, add CI Dead code (AUDIT 2.3 and beyond): - app.py: `_analysistab_default` instantiated `Analysis`, whose import was commented out and for which no trait exists -- a latent NameError. Removed along with the commented-out import and two commented Instance declarations. - app.py: ApplicationMain.__init__ never called HasTraits.__init__, so trait initialisation was skipped and kwargs were silently discarded. - halos.py: `boxtype` was declared twice, the second silently winning. - Three unused `modules.mergertrees` imports (only the merger tree tab uses it), five unused Button traits wired to no handler and no view, `PBSstring` and `PBSjobname`, and ~40 lines of commented-out sample lists and debug prints. - 20 dead local assignments across four tabs, several of which existed only to feed commented-out debug prints. Removing `xpossub`/`mhalfsub` and friends in the contamination tab cascaded: the sub-halo sort they fed was itself unread. - The commented-out PBS example string embedding a personal absolute path. Duplication: - mergertree.py: four byte-identical `_*_changed` handlers become one `_invalidate_plot`, each of which also fetched an unused `ax`. File handling: - No raw `open()` remains outside cme.writers. The Gadget driver script is accumulated in a list and written once rather than held open across the whole sweep, and the two Lagrangian output files move to writers.lagrangian_header / writers.lagrangian_region. - Two `try: open(path)` existence checks become os.path.exists, and the one that reported failure via print() now sets the status field. Style: 17 `== True` / `== False` comparisons simplified (leaving the `enabled_when` strings alone -- those are Traits expressions, not Python), tabs in install.py converted to spaces, and printf-style formatting modernised. CI runs pytest, doctests, ruff check and a format check on three Python versions, plus compileall over the GUI modules -- which cannot be imported without a display or the absent `modules` package, but must at least parse. `ruff check .` is clean; the format check is scoped to the display-free modules and tests, since reflowing the TraitsUI view declarations would bury this branch's behavioural changes under whitespace. --- .github/workflows/ci.yml | 89 ++++++ pyproject.toml | 17 +- src/cme/__init__.py | 1 - src/cme/app.py | 28 +- src/cme/config.py | 22 +- src/cme/cosmology.py | 58 +++- src/cme/naming.py | 14 +- src/cme/science/candidates.py | 14 +- src/cme/shell.py | 5 +- src/cme/ui/candidates.py | 43 ++- src/cme/ui/common.py | 32 +- src/cme/ui/contamination.py | 52 +--- src/cme/ui/gadget.py | 111 ++++--- src/cme/ui/halos.py | 44 +-- src/cme/ui/home.py | 3 +- src/cme/ui/ics.py | 126 ++++---- src/cme/ui/install.py | 10 +- src/cme/ui/mergertree.py | 68 ++--- src/cme/writers/__init__.py | 13 + src/cme/writers/gadget.py | 501 +++++++++++++++++++------------ src/cme/writers/music.py | 18 +- src/cme/writers/submit.py | 4 +- tests/test_config.py | 15 +- tests/test_cosmology.py | 10 +- tests/test_naming.py | 72 +++-- tests/test_science_candidates.py | 172 ++++++----- tests/test_writers_gadget.py | 100 ++++-- tests/test_writers_music.py | 18 +- tests/test_writers_submit.py | 30 +- 29 files changed, 978 insertions(+), 712 deletions(-) create mode 100644 .github/workflows/ci.yml diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml new file mode 100644 index 0000000..021f42d --- /dev/null +++ b/.github/workflows/ci.yml @@ -0,0 +1,89 @@ +name: CI + +on: + push: + branches: [master] + pull_request: + +jobs: + test: + runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + python-version: ["3.9", "3.11", "3.12"] + + steps: + - uses: actions/checkout@v4 + + - uses: actions/setup-python@v5 + with: + python-version: ${{ matrix.python-version }} + cache: pip + + # Only the display-free modules are installed and tested. The GUI extra + # needs wxPython, Mayavi and a display, and the tabs additionally import + # the `modules` simulation-I/O package, which is not distributed with this + # repository -- see README.md. + - name: Install + run: | + python -m pip install --upgrade pip + python -m pip install -e ".[dev]" + + - name: Test + run: python -m pytest tests/ -v + + - name: Doctests + run: | + python -m pytest --doctest-modules \ + src/cme/naming.py \ + src/cme/cosmology.py \ + src/cme/config.py \ + src/cme/writers/music.py \ + src/cme/writers/gadget.py \ + src/cme/writers/submit.py \ + src/cme/science/candidates.py + + lint: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: "3.12" + cache: pip + + - name: Install ruff + run: python -m pip install ruff + + - name: Lint + run: ruff check . + + # Scoped to the display-free modules and the tests. The TraitsUI tabs are + # not yet formatted: reflowing those nested view declarations would bury + # the behavioural changes in this branch under whitespace. + - name: Format check + run: | + ruff format --check \ + src/cme/naming.py \ + src/cme/cosmology.py \ + src/cme/shell.py \ + src/cme/config.py \ + src/cme/writers/ \ + src/cme/science/ \ + tests/ + + syntax: + # The GUI modules cannot be imported without a display or the `modules` + # package, but they must at least parse on every supported version. + runs-on: ubuntu-latest + strategy: + matrix: + python-version: ["3.9", "3.12"] + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: ${{ matrix.python-version }} + - name: Compile all modules + run: python -m compileall -q src/ diff --git a/pyproject.toml b/pyproject.toml index 8dce8ab..a065f08 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -61,10 +61,19 @@ src = ["src", "tests"] [tool.ruff.lint] select = ["E", "F", "W", "I", "UP", "B", "C4", "SIM"] -# The GUI modules are wide TraitsUI view declarations where a nested call -# chain reads better than a wrapped one; they are excluded from line-length -# and star-import rules until they are broken up. -ignore = ["E501"] +ignore = [ + # TraitsUI view declarations are wide nested call chains where wrapping to + # 100 columns hurts more than it helps. + "E501", + # dict(a=1) reads better than {"a": 1} for keyword-style parameter sets, + # which is how the run-parameter dicts and test fixtures are written. + "C408", +] [tool.ruff.lint.per-file-ignores] +# The GUI modules and the app entry point share state through +# `from cme.ui.common import *`. Unpicking that means giving every tab an +# explicit import list; until then these two rules cannot fire usefully. +# `common.py` itself declares __all__, so the surface is at least explicit. "src/cme/ui/*.py" = ["F403", "F405"] +"src/cme/app.py" = ["F403", "F405"] diff --git a/src/cme/__init__.py b/src/cme/__init__.py index c1c26ed..f42c808 100644 --- a/src/cme/__init__.py +++ b/src/cme/__init__.py @@ -1,4 +1,3 @@ -# coding: utf-8 """ A tool for constructing and analysing cosmological simulations. """ diff --git a/src/cme/app.py b/src/cme/app.py index 1716e8d..cb360f0 100644 --- a/src/cme/app.py +++ b/src/cme/app.py @@ -14,7 +14,6 @@ from cme.ui.install import InstallFiles from cme.ui.mergertree import MergerTree -#from analysis import Analysis class ApplicationMain(HasTraits): @@ -22,27 +21,24 @@ class ApplicationMain(HasTraits): headertab = Instance(Header) installtab = Instance(InstallFiles) - #existencetab = Instance(Existence) initstab = Instance(InitialConditions) gadgetruntab = Instance(GadgetRun) halofindtab = Instance(HaloFind) mergertreetab = Instance(MergerTree) candidatestab = Instance(Candidates) contaminationtab = Instance(Contamination) - #analysistab = Instance(Analysis) display = Instance(Figure) markercolor = ColorTrait markerstyle = Enum(['+',',','*','s','p','d','o']) markersize = Range(0,10,2) - + left_panel = Tabbed(Group(VGroup(Item('display', editor=MPLFigureEditor(),show_label=False, resizable=True)), HGroup(Item(name='markercolor', label="Color", style="custom",springy=True), Item(name='markerstyle', label="Marker",springy=True), Item(name='markersize', label="Size",springy=True)), label='Display'), Item(name='scene',label='Mayavi',editor=SceneEditor(scene_class=MayaviScene)),show_labels=False) -# right_panel = Tabbed(Item('headertab', style='custom', label='Home',show_label=False), Item('installtab', style='custom', label='Install',show_label=False), Item('candidatestab', style='custom', label='Candidates',show_label=False), @@ -51,8 +47,7 @@ class ApplicationMain(HasTraits): Item('halofindtab', style='custom', label='Halo Finder',show_label=False), Item('mergertreetab', style='custom', label='Merger Tree',show_label=False), Item('contaminationtab', style='custom', label='Contam.',show_label=False)) - #Item('analysistab', style='custom', label='Analysis',show_label=False)) - + view = View(HSplit(left_panel, right_panel), width = 1280, @@ -70,7 +65,7 @@ def _display_default(self): ax.set_ylabel('Y') ax.set_xlim(0,1) ax.set_ylim(0,1) - + # Set matplotlib canvas colour to be white rect = figure.patch rect.set_facecolor('w') @@ -96,35 +91,32 @@ def _halofindtab_default(self): def _mergertreetab_default(self): return MergerTree(self) - def _analysistab_default(self): - return Analysis(self) - def _candidatestab_default(self): return Candidates(self) - + def _contaminationtab_default(self): return Contamination(self) def _markercolor_changed(self): - ax = self.display.axes[0] - if hasattr(self, 'display_points'): + if hasattr(self, 'display_points'): self.display_points.set_color(self.markercolor) self.display_points.set_markeredgecolor(self.markercolor) wx.CallAfter(self.display.canvas.draw) def _markerstyle_changed(self): - ax = self.display.axes[0] - if hasattr(self, 'display_points'): + if hasattr(self, 'display_points'): self.display_points.set_marker(self.markerstyle) wx.CallAfter(self.display.canvas.draw) def _markersize_changed(self): - ax = self.display.axes[0] - if hasattr(self, 'display_points'): + if hasattr(self, 'display_points'): self.display_points.set_markersize(self.markersize) wx.CallAfter(self.display.canvas.draw) def __init__(self, **kwargs): + # HasTraits.__init__ was previously never called, so trait + # initialisation was skipped and any kwargs were silently dropped. + HasTraits.__init__(self, **kwargs) self.markercolor = 'blue' self.markersize = 2 self.markerstyle = 'o' diff --git a/src/cme/config.py b/src/cme/config.py index ec51330..5609b38 100644 --- a/src/cme/config.py +++ b/src/cme/config.py @@ -48,8 +48,10 @@ def config_path(): candidates = [ os.path.expanduser("~/.config/cme/clusters.toml"), - os.path.join(os.path.dirname(os.path.dirname(os.path.dirname( - os.path.abspath(__file__)))), "clusters.toml"), + os.path.join( + os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))), + "clusters.toml", + ), os.path.join(os.path.dirname(os.path.abspath(__file__)), "clusters.toml"), ] for path in candidates: @@ -95,19 +97,21 @@ def detect(config=None, hostname=None): def resolve(name=None, config=None, username=None): """Return one profile's paths with placeholders expanded. - >>> paths = resolve("local", username="alice") - >>> paths["master"].endswith("alice") - True - Keys: ``name``, ``username``, ``home``, ``master``, plus the entries in :data:`PATH_KEYS` and any scheduler settings the profile carries. + + A profile with a ``home_root`` treats it as the directory containing + per-user homes, so the user's own directory sits below it. Without one, the + expanded home directory already identifies the user and the name is not + appended: + + >>> resolve("local", username="alice")["master"] == os.path.expanduser("~") + True """ config = config or load() name = name or detect(config) if name not in config["clusters"]: - raise ValueError( - f"unknown cluster {name!r}; expected one of {profile_names(config)}" - ) + raise ValueError(f"unknown cluster {name!r}; expected one of {profile_names(config)}") profile = config["clusters"][name] user = username or profile.get("username") or getpass.getuser() diff --git a/src/cme/cosmology.py b/src/cme/cosmology.py index eed178c..80d5b4e 100644 --- a/src/cme/cosmology.py +++ b/src/cme/cosmology.py @@ -58,45 +58,73 @@ def as_tuple(self): WMAP1 = Cosmology( - name="WMAP1", omega_m=0.270, omega_l=0.730, omega_b=0.0440, - h=0.710, sigma_8=0.840, n_spec=0.990, + name="WMAP1", + omega_m=0.270, + omega_l=0.730, + omega_b=0.0440, + h=0.710, + sigma_8=0.840, + n_spec=0.990, reference="Spergel et al. 2003, ApJS 148, 175", ) WMAP3 = Cosmology( - name="WMAP3", omega_m=0.234, omega_l=0.766, omega_b=0.0407, - h=0.734, sigma_8=0.760, n_spec=0.961, + name="WMAP3", + omega_m=0.234, + omega_l=0.766, + omega_b=0.0407, + h=0.734, + sigma_8=0.760, + n_spec=0.961, reference="Spergel et al. 2007, ApJS 170, 377", ) WMAP5 = Cosmology( - name="WMAP5", omega_m=0.258, omega_l=0.742, omega_b=0.0441, - h=0.719, sigma_8=0.796, n_spec=0.963, + name="WMAP5", + omega_m=0.258, + omega_l=0.742, + omega_b=0.0441, + h=0.719, + sigma_8=0.796, + n_spec=0.963, reference="Komatsu et al. 2009, ApJS 180, 330", ) WMAP7 = Cosmology( - name="WMAP7", omega_m=0.272, omega_l=0.728, omega_b=0.0455, - h=0.704, sigma_8=0.807, n_spec=0.961, + name="WMAP7", + omega_m=0.272, + omega_l=0.728, + omega_b=0.0455, + h=0.704, + sigma_8=0.807, + n_spec=0.961, reference="Komatsu et al. 2011, ApJS 192, 18", ) WMAP9 = Cosmology( - name="WMAP9", omega_m=0.2865, omega_l=0.7135, omega_b=0.0463, - h=0.6932, sigma_8=0.820, n_spec=0.9608, + name="WMAP9", + omega_m=0.2865, + omega_l=0.7135, + omega_b=0.0463, + h=0.6932, + sigma_8=0.820, + n_spec=0.9608, reference="Hinshaw et al. 2013, ApJS 208, 19", ) PLANCK = Cosmology( - name="PLANCK", omega_m=0.3175, omega_l=0.6825, omega_b=0.0490, - h=0.6711, sigma_8=0.8344, n_spec=0.9624, + name="PLANCK", + omega_m=0.3175, + omega_l=0.6825, + omega_b=0.0490, + h=0.6711, + sigma_8=0.8344, + n_spec=0.9624, reference="Planck Collaboration 2014, A&A 571, A16 (Planck 2013 XVI)", ) #: Keyed by the names the GUI's CheckListEditor offers. -COSMOLOGIES = { - c.name: c for c in (WMAP1, WMAP3, WMAP5, WMAP7, WMAP9, PLANCK) -} +COSMOLOGIES = {c.name: c for c in (WMAP1, WMAP3, WMAP5, WMAP7, WMAP9, PLANCK)} #: The Caterpillar suite was run with Planck 2013. Tabs that analyse existing #: output and have no cosmology selector of their own assume this, which is diff --git a/src/cme/naming.py b/src/cme/naming.py index 596bc81..2f7f256 100644 --- a/src/cme/naming.py +++ b/src/cme/naming.py @@ -111,9 +111,7 @@ def run_dir(base, halo_id, **kwargs): ``kwargs`` are forwarded to :func:`run_dir_name`. """ - return os.path.join( - halo_dir(base, halo_id), run_dir_name(halo_id=halo_id, **kwargs) - ) + return os.path.join(halo_dir(base, halo_id), run_dir_name(halo_id=halo_id, **kwargs)) def outputs_dir(base, halo_id, **kwargs): @@ -131,11 +129,7 @@ def job_name(halo_id, region, padding, levelmax, nrvir): 'H19EP7L11N4' """ return ( - f"H{str(int(halo_id))[:2]}" - f"{region_code(region)}" - f"P{int(padding)}" - f"L{int(levelmax)}" - f"N{int(nrvir)}" + f"H{str(int(halo_id))[:2]}{region_code(region)}P{int(padding)}L{int(levelmax)}N{int(nrvir)}" ) @@ -164,9 +158,7 @@ def snapshot_dir(outputs, snapnum): def snapshot_base(outputs, snapnum): """Path stem the Gadget readers expect for a snapshot (no file suffix).""" - return os.path.join( - snapshot_dir(outputs, snapnum), f"snap_{snapshot_index(snapnum)}" - ) + return os.path.join(snapshot_dir(outputs, snapnum), f"snap_{snapshot_index(snapnum)}") def groups_dir(outputs, snapnum): diff --git a/src/cme/science/candidates.py b/src/cme/science/candidates.py index a3ba289..a63960f 100644 --- a/src/cme/science/candidates.py +++ b/src/cme/science/candidates.py @@ -71,9 +71,7 @@ def _min_distance(x, y, z, xs, ys, zs, hubble, exclude=None): xs, ys, zs = xs[exclude], ys[exclude], zs[exclude] if xs.size == 0: return np.inf - return float( - np.sqrt((x - xs) ** 2 + (y - ys) ** 2 + (z - zs) ** 2).min() / hubble - ) + return float(np.sqrt((x - xs) ** 2 + (y - ys) ** 2 + (z - zs) ** 2).min() / hubble) def select(hosts, mass_range, zones, hubble): @@ -153,9 +151,13 @@ def to_text(candidates): lines = ["# ID M Rvir x y z"] for row in candidates: lines.append( - "%f %e %f %f %f %f " % ( - int(row["id"]), row["mvir"], row["rvir"], - row["posX"], row["posY"], row["posZ"], + "{:f} {:e} {:f} {:f} {:f} {:f} ".format( + int(row["id"]), + row["mvir"], + row["rvir"], + row["posX"], + row["posY"], + row["posZ"], ) ) return "\n".join(lines) + "\n" diff --git a/src/cme/shell.py b/src/cme/shell.py index 2b9e8db..644f3c3 100644 --- a/src/cme/shell.py +++ b/src/cme/shell.py @@ -49,8 +49,9 @@ def run_all(commands, cwd=None, check=False): result = run(args, cwd=cwd, check=check) results.append(result) if result.returncode != 0: - log.warning("command failed (%s), stopping: %s", - result.returncode, [str(a) for a in args]) + log.warning( + "command failed (%s), stopping: %s", result.returncode, [str(a) for a in args] + ) break return results diff --git a/src/cme/ui/candidates.py b/src/cme/ui/candidates.py index d4be24f..dea1e29 100644 --- a/src/cme/ui/candidates.py +++ b/src/cme/ui/candidates.py @@ -1,8 +1,7 @@ -from cme.ui.common import * - from cme import cosmology, writers from cme.science import candidates as sci -from numpy.random import random +from cme.ui.common import * + class Candidates(HasTraits): @@ -44,16 +43,12 @@ class Candidates(HasTraits): nlargerthanuppermasscut = Int nbetweenmasscut = Int ntotalcandidates = Int - upper = Float(15.) - + candidatearr = Array loadpreviouscand = Bool(True) - - #candidatearr = random((20, 6)) + haloid = List(Int) - #haloid = Enum(['190897','208737','140666','28221','147419','28188','147273','78411','131988','19910']) - #haloid = Enum() jobstatus = Str() addhalo_button = Button("ADD") haloidtoadd = Int() @@ -62,14 +57,14 @@ class Candidates(HasTraits): view = View(Tabbed(Group(Group(Item(name='halopath',label='Parent Simulation Path'), HGroup(Item(name='loadpreviouscand',label='Use Existing Candidates'), Item(name='exclusionstr',show_label=False,style='readonly')), - + Group(HGroup(Item(name='lowermasscut',label='log[M > Msol]',springy=True,format_str= '%.1f'), Item(name='normlowermasscut',label='M',width=-70,format_str='%.2e')), HGroup(Item(name='uppermasscut',label='log[M > Msol]',springy=True,format_str= '%.1f'), Item(name='normuppermasscut',label='M',width=-70,format_str='%.2e')) ,label='Mass Range For Base Sample',show_border=True,enabled_when='loadpreviouscand == False'), - - + + Group(Group(Group(HGroup(Item(name='largerthan1',label='log[M > Msol]',springy=True), Item(name='normlargerthan1',label='M',width=-70,format_str='%.2e')),enabled_when='useintmult1==False'), @@ -77,21 +72,21 @@ class Candidates(HasTraits): Group(Item(name='moremassivethancand1',label='N',springy=True),enabled_when='useintmult1==True',springy=True)), Item(name='distancecut1',label='Within [Mpc]'),label='Exclusion Zone #1',show_border=True),enabled_when='loadpreviouscand == False'), - + Group(Group(Group(HGroup(Item(name='largerthan2',label='log[M > Msol]',springy=True), Item(name='normlargerthan2',label='M',width=-70,format_str='%.2e')),enabled_when='useintmult2==False'), HGroup(Item(name='useintmult2',label='Use N times candidates mass'), Group(Item(name='moremassivethancand2',label='N',springy=True),enabled_when='useintmult2==True',springy=True)), Item(name='distancecut2',label='Within [Mpc]'),label='Exclusion Zone #2',show_border=True),enabled_when='loadpreviouscand == False'), - + Group(Group(Group(HGroup(Item(name='largerthan3',label='log[M > Msol]',springy=True), Item(name='normlargerthan3',label='M',width=-70,format_str='%.2e')),enabled_when='useintmult3==False'), HGroup(Item(name='useintmult3',label='Use N times candidates mass'), Group(Item(name='moremassivethancand3',label='N',springy=True),enabled_when='useintmult3==True',springy=True)), Item(name='distancecut3',label='Within [Mpc]'),label='Exclusion Zone #3',show_border=True),enabled_when='loadpreviouscand == False'), - + HGroup(Item(name='writeoutopt',label='W/O?'), HGroup(Item(name='writeoutpath',label='Directory',springy=True), Item(name='writeoutname',label='Filename',springy=True),enabled_when='writeoutopt == True',springy=True), @@ -120,8 +115,6 @@ class Candidates(HasTraits): def _haloid_default(self): return [268422,242183,140666,241932,96624,121869,1930,173411,21047,233776] - #[190897,208737] - #,140666,28221,147419,28188,147273,78411,131988,19910] def _addhalo_button_fired(self): # Previously `!= 1`, which tested for the phantom zero row that used to @@ -198,7 +191,7 @@ def _plot_candidates(self): def _getcandidates_button_fired(self): hubble = cosmology.DEFAULT.h - if self.loadpreviouscand == False: + if not self.loadpreviouscand: halodata = RSDataReader.RSDataReader(self.halopath, self.snapnum, digits=2) hosts = halodata.get_hosts() @@ -216,14 +209,14 @@ def _getcandidates_button_fired(self): [found[name] for name in sci.COLUMNS] ) if len(found) else np.zeros(shape=(0, 6)) - if self.writeoutopt == True: + if self.writeoutopt: writers.write(os.path.join(self.writeoutpath, self.writeoutname), sci.to_text(found)) if len(found): self._plot_candidates() - elif self.loadpreviouscand == True: + elif self.loadpreviouscand: cand = getcandidatelist(os.path.join(self.writeoutpath, self.writeoutname)) self.candidatearr = np.asarray(cand)[:, :6] self.ntotalcandidates = len(self.candidatearr) @@ -236,7 +229,7 @@ def _largerthan2_changed(self): self.normlargerthan2 = 10**self.largerthan2 def _largerthan3_changed(self): - self.normlargerthan3 = 10**self.largerthan3 + self.normlargerthan3 = 10**self.largerthan3 def _normlargerthan1_changed(self): self.largerthan1 = np.log10(self.normlargerthan1) @@ -252,20 +245,19 @@ def _uppermasscut_changed(self): def _normuppermasscut_changed(self): self.uppermasscut = np.log10(self.normuppermasscut) - + def _lowermasscut_changed(self): self.normlowermasscut = 10**self.lowermasscut def _normlowermasscut_changed(self): self.lowermasscut = np.log10(self.normlowermasscut) - + def __init__(self, main, **kwargs): HasTraits.__init__(self) self.main = main self.exclusionstr = 'Exclusion Zones: "no halo with mass M within distance R"' self.hubble = cosmology.DEFAULT.h - #self.haloid = [190897] self.lowermasscut = np.log10(7e11) self.uppermasscut = np.log10(7e12) @@ -290,8 +282,7 @@ def __init__(self, main, **kwargs): self.normlargerthan1 = 10**self.largerthan1 self.normlargerthan2 = 10**self.largerthan2 self.normlargerthan3 = 10**self.largerthan3 - #self.largerthan3 = np.log10(7e13) - + self.writeoutname = 'candidates.dat' self.writeoutpath = self.main.headertab.datamasterpath self.halopath = self.main.headertab.parentsimpath + 'RockstarData' diff --git a/src/cme/ui/common.py b/src/cme/ui/common.py index 801cd28..5bc4537 100644 --- a/src/cme/ui/common.py +++ b/src/cme/ui/common.py @@ -9,14 +9,27 @@ import platform import subprocess +import modules.readhalos.readsubf as readsubf +import modules.readhalos.RSDataReaderv2 as RSDataReader +import modules.readsnapshots.readsnap as rs +import modules.readsnapshots.readsnapHDF5 as rsHD import numpy as np import wx +from enable.api import ColorTrait from matplotlib import patches from matplotlib.backends.backend_wx import NavigationToolbar2Wx from matplotlib.backends.backend_wxagg import FigureCanvasWxAgg as FigureCanvas from matplotlib.figure import Figure - -from enable.api import ColorTrait +from modules.brendanlib.grifflib import ( + COM, + CorrectPos, + addsubtitle, + drawcircle, + getcandidatelist, + getcentext, + getlagrxyz, + placenormtext, +) from traits.api import ( Array, Bool, @@ -47,21 +60,6 @@ from traitsui.ui_editors.array_view_editor import ArrayViewEditor from traitsui.wx.editor import Editor -import modules.readhalos.RSDataReaderv2 as RSDataReader -import modules.readhalos.readsubf as readsubf -import modules.readsnapshots.readsnap as rs -import modules.readsnapshots.readsnapHDF5 as rsHD -from modules.brendanlib.grifflib import ( - COM, - CorrectPos, - addsubtitle, - drawcircle, - getcandidatelist, - getcentext, - getlagrxyz, - placenormtext, -) - __all__ = [ # Standard library and numerics "os", diff --git a/src/cme/ui/contamination.py b/src/cme/ui/contamination.py index 11e881a..114b73b 100644 --- a/src/cme/ui/contamination.py +++ b/src/cme/ui/contamination.py @@ -1,14 +1,10 @@ -from cme.ui.common import * -import modules.mergertrees.MTCatalogue as MT - from cme import cosmology, naming +from cme.ui.common import * class Contamination(HasTraits): gadpath = Directory - checkexistence_button = Button("Check Existence") - createconfig_button = Button("Make and distribute gadget!") checkcontam_button = Button("Plot Radial Distribution") plothalodist_button = Button("Plot Heatmap") @@ -32,8 +28,6 @@ class Contamination(HasTraits): hostposy = Float() hostposz = Float() hostmass = Float() - vizexpz_button = Button("Inspect/Update List") - subscript_button = Button("Create Submission Script") includegroupsopt = Bool(False) xvar = Enum(['X-POS','Y-POS','Z-POS']) yvar = Enum(['X-POS','Y-POS','Z-POS']) @@ -46,7 +40,7 @@ class Contamination(HasTraits): yref = Float(50.) zref = Float(50.) - + view = View(Item('gadpath',label='Base Path'), Item(name='haloid',label='Halo ID'), @@ -91,7 +85,6 @@ def _checkcontam_button_fired(self): s = readsubf.subfind_catalog(tmppath, self.snapnum) #ids = readsubf.subf_ids(tmppath, self.snapnum, 0, 0, read_all=1) mgroup = s.group_m_mean200*10**10/hubble - rvirgroup = s.group_r_mean200 xposgroup = s.group_pos[:,0] yposgroup = s.group_pos[:,1] zposgroup = s.group_pos[:,2] @@ -114,11 +107,6 @@ def _checkcontam_button_fired(self): zposhost = zposgroup[cond] mvir = mgroup[cond] - rvirsub = s.sub_halfmassrad - xpossub = s.sub_pos[:,0] - ypossub = s.sub_pos[:,1] - zpossub = s.sub_pos[:,2] - mhalfsub = s.sub_mass*10**10/hubble #print s.sub_grnr titlestr = "Type: " + str(self.boxtype[0]) + "\n" + \ @@ -126,17 +114,7 @@ def _checkcontam_button_fired(self): "n*rvir(z=0): " + str(self.nrvir) + "\n" + \ "level max: " + str(self.lmax) + "\n" - - R = np.sqrt((self.xref-xpossub)**2+(self.yref-ypossub)**2+(self.zref-zpossub)**2) - sortindex = np.argsort(R) - Rsorted = R[sortindex] - Msorted = mhalfsub[sortindex] - Groupsorted = s.sub_grnr[sortindex] - masscon = (Msorted > self.lowermassrange) & (Msorted < self.uppermassrange) & (Rsorted < self.deltar) - #print Groupsorted[masscon] - #print Rsorted[masscon] - #print Msorted[masscon] if len(xposhost) > 1: self.datastatus = "Too many possible hosts found, be more strict." @@ -156,13 +134,13 @@ def _checkcontam_button_fired(self): self.hostposy = float(yposhost) self.hostposz = float(zposhost) self.hostmass = float(mvir) - + R = np.sqrt((xposhost-xposgroup)**2+(yposhost-yposgroup)**2+(zposhost-zposgroup)**2) sortindex = np.argsort(R) Rsorted = R[sortindex] contamNR = np.cumsum(s.group_contamination_count[sortindex]) contamMR = np.cumsum(s.group_contamination_mass[sortindex]*10**10) - + figure = self.main.display figure.clear() ax = figure.add_subplot(111) @@ -172,12 +150,12 @@ def _checkcontam_button_fired(self): ax.plot(Rsorted,np.log10(np.array(contamNR)),linestyle='-',color='b',linewidth=2) ax.set_xlim([0,7]) ax.set_ylim([0,5]) - + ax.set_ylabel(r'$\mathrm{\Sigma\ log_{10}\ N_{CP}}$', color='b',fontsize=14) - + for tl in ax.get_yticklabels(): tl.set_color('b') - + axb = ax.twinx() axb.tick_params(axis='both', which='major', labelsize=12) axb.set_ylim([8,14]) @@ -185,12 +163,12 @@ def _checkcontam_button_fired(self): axb.set_ylim([8,14]) for tl in axb.get_yticklabels(): tl.set_color('r') - + axb.set_ylabel(r'$\mathrm{log_{10}\ \Sigma\ M_{CP}\ [M_\odot/h]}$', color='r',fontsize=14) ax.set_xlabel(r'$\mathrm{R_{FOF}\ [Mpc/h]}$',fontsize=14) #ax.set_ylabel(r'$\mathrm{y-pos\ [Mpc/h]}$') #ax.set_xlabel(r'$\mathrm{y-pos\ [Mpc/h]}$') - + wx.CallAfter(self.main.display.canvas.draw) else: @@ -284,7 +262,7 @@ def _plothalodist_button_fired(self): xposmgroup = s.group_pos[:,0] yposmgroup = s.group_pos[:,1] zposmgroup = s.group_pos[:,2] - + if self.weightopt == 'count': weights = s.group_contamination_count @@ -319,10 +297,10 @@ def _plothalodist_button_fired(self): heatmap, xedges, yedges = np.histogram2d(xvar, yvar, bins=self.nbins,weights=weights) extent = [xedges[0], xedges[-1], yedges[0], yedges[-1]] - - if self.includegroupsopt == True: - xcirc,ycirc = drawcircle(xvar[mgroup > 10**self.lowerlogmassbound],yvar[mgroup > 10**self.lowerlogmassbound],rvirmgroup[mgroup > 10**self.lowerlogmassbound]) - ax.plot(xcirc,ycirc,'k-',linewidth=2,alpha=0.3) + + if self.includegroupsopt: + xcirc,ycirc = drawcircle(xvar[mgroup > 10**self.lowerlogmassbound],yvar[mgroup > 10**self.lowerlogmassbound],rvirmgroup[mgroup > 10**self.lowerlogmassbound]) + ax.plot(xcirc,ycirc,'k-',linewidth=2,alpha=0.3) heatmap = np.flipud(np.rot90(heatmap)) sc1 = ax.imshow(np.log10(heatmap),extent = extent,cmap = 'jet', origin='lower') @@ -381,4 +359,4 @@ def __init__(self, main, **kwargs): self._refresh() - + diff --git a/src/cme/ui/gadget.py b/src/cme/ui/gadget.py index 014c73c..9df2c7e 100644 --- a/src/cme/ui/gadget.py +++ b/src/cme/ui/gadget.py @@ -1,21 +1,19 @@ -from cme.ui.common import * - import shutil from cme import cosmology, naming, shell, writers -from cme.writers import gadget as gadgetwriter, submit +from cme.ui.common import * +from cme.writers import gadget as gadgetwriter +from cme.writers import submit #: Driver script the tab writes and then runs. DRIVER_SCRIPT = "rungadget.sh" -import modules.mergertrees.MTCatalogue as MT class GadgetRun(HasTraits): #clusteropt = List() - + PBSncores = Range(0,64,8) PBSnnodes = Range(0,10,8) username = Str() - PBSjobname = Str() PBSqueue = Str("default") outfile = Str("OUTPUT") errorfile = Str("ERROR") @@ -102,7 +100,6 @@ class GadgetRun(HasTraits): gadpath = Directory checkexistence_button = Button("Check Existence") - createconfig_button = Button("Make and distribute gadget!") padding = List(editor = CheckListEditor(values = ['5','6','7','8','9','10'],cols=6) ) lmin = Enum(['7','8','9','10','11','12','13','14','15']) lmax = Enum(['7','8','9','10','11','12','13','14','15']) @@ -118,7 +115,6 @@ class GadgetRun(HasTraits): nintervals = Enum(['32','64','128','258','512','1024']) snapshotlist = Str() alloutputopt = Bool(True) - PBSstring = Str() vizexpz_button = Button("Inspect/Update List") subscript_button = Button("Run Script") submitjob = Bool(False) @@ -170,7 +166,7 @@ class GadgetRun(HasTraits): Item('BufferSize',label='BufferSize'), Item('TimeBegin',label='TimeBegin'), Item('TimeMax',label='TimeMax'),label='Format'), - + Group(Item('ComovingIntegrationOn',label='ComovingIntegrationOn'), Item('PeriodicBoundariesOn',label='PeriodicBoundariesOn'), Item('CoolingOn',label='CoolingOn'), @@ -205,7 +201,7 @@ class GadgetRun(HasTraits): Item('UnitMass_in_g',label='UnitMass_in_g'), Item('UnitVelocity_in_cm_per_s',label='UnitVelocity_in_cm_per_s'), Item('GravityConstantInternal',label='GravityConstantInternal'), - + Item('MinGasHsmlFractional',label='MinGasHsmlFractional'), Item('SofteningGas',label='SofteningGas'), Item('SofteningHalo',label='SofteningHalo'), @@ -256,15 +252,17 @@ class GadgetRun(HasTraits): Item('SLURMmemory',label='Memory'), Item('SLURMqueue',label='Queue'), Item('SLURMexecute',label='Execute Line'),show_border=True,enabled_when="subtype=='SLURM'"), - + HGroup(Item('compilegadopt',label='(Re-)Compile Gadget?'),Item('submitjob',label='Submit Job?'),Item('subscript_button',show_label=False)), label='Write & Submit Job') ) def _subscript_button_fired(self): - f1 = open(DRIVER_SCRIPT, 'w') - f1.write("#!/bin/bash \n") + # Accumulated and written once, so the handle cannot leak on an + # exception part-way through the sweep. + lines = [] + lines.append("#!/bin/bash \n") # The Gadget source tree comes from the Home tab, which derives it from the # selected cluster profile. This used to be two hardcoded absolute paths @@ -274,12 +272,12 @@ def _subscript_button_fired(self): if self.subtype == "SLURM": for line in self.modulelines.splitlines(): - f1.write(line + "\n") + lines.append(line + "\n") - f1.write("cd " + gadgetpath + "\n") + lines.append("cd " + gadgetpath + "\n") - f1.write("make clean\n") - f1.write("make -j 8\n") + lines.append("make clean\n") + lines.append("make -j 8\n") for boxtypei in self.boxtype: for nrviri in self.nrvir: @@ -288,7 +286,6 @@ def _subscript_button_fired(self): run = dict(region=boxtypei, padding=paddingi, levelmin=self.lmin, levelmax=self.lmax, overlap=overlapi, nrvir=nrviri, zinit=self.zinit) - foldername = naming.run_dir_name(halo_id=self.haloid, **run) jobname = naming.job_name(halo_id=self.haloid, region=boxtypei, padding=paddingi, levelmax=self.lmax, nrvir=nrviri) @@ -304,13 +301,13 @@ def _subscript_button_fired(self): # and then shelled out to `mv`. self.makeparam(os.path.join(filepath, 'param.txt'), filepath, includesub=False) - if self.ENABLE_SUBFIND == True: + if self.ENABLE_SUBFIND: self.makeparam(os.path.join(filepath, 'param_sub.txt'), filepath, includesub=True) os.makedirs(os.path.join(filepath, "outputs"), exist_ok=True) - + scriptpath = os.path.join( filepath, "runscript_sub" if self.ENABLE_SUBFIND else "runscript") @@ -339,7 +336,7 @@ def _subscript_button_fired(self): extra_lines=self.PBSextralines) writers.write(scriptpath, script) - if self.compilegadopt == True: + if self.compilegadopt: name = ("Config_sub.sh" if self.ENABLE_SUBFIND else "Config.sh") self.install_config_sh( @@ -348,35 +345,34 @@ def _subscript_button_fired(self): "Config.sh")) #try: # with open("/bigbang/data/bgriffen/lib/P-Gadget3/P-Gadget3"): pass - f1.write("cd " + str(filepath) + "\n") - f1.write("rm " + str(filepath) + "/*.e*\n") - f1.write("rm " + str(filepath) + "/*.o*\n") - #f1.write("rm " + str(filepath) + "/ExpansionList_64\n") - - if self.ENABLE_SUBFIND == True: - f1.write("\n") - f1.write("rm P-Gadget3_sub\n") - f1.write("rm P-Gadget3_Sub\n") - f1.write("rm ERRORsub\n") - f1.write("rm OUTPUTsub\n") - f1.write("cp " + gadgetfilepath + "P-Gadget3/P-Gadget3 " + filepath + "/P-Gadget3_sub\n") - if self.submitjob == True: - f1.write("qsub runscript_sub \n") + lines.append("cd " + str(filepath) + "\n") + lines.append("rm " + str(filepath) + "/*.e*\n") + lines.append("rm " + str(filepath) + "/*.o*\n") + #lines.append("rm " + str(filepath) + "/ExpansionList_64\n") + + if self.ENABLE_SUBFIND: + lines.append("\n") + lines.append("rm P-Gadget3_sub\n") + lines.append("rm P-Gadget3_Sub\n") + lines.append("rm ERRORsub\n") + lines.append("rm OUTPUTsub\n") + lines.append("cp " + gadgetfilepath + "P-Gadget3/P-Gadget3 " + filepath + "/P-Gadget3_sub\n") + if self.submitjob: + lines.append("qsub runscript_sub \n") else: - f1.write("\n") - f1.write("rm P-Gadget3\n") - f1.write("rm ERROR\n") - f1.write("rm OUTPUT\n") - f1.write("cp " + gadgetfilepath + "P-Gadget3/P-Gadget3 " + filepath + "/P-Gadget3\n") - - if self.submitjob == True: + lines.append("\n") + lines.append("rm P-Gadget3\n") + lines.append("rm ERROR\n") + lines.append("rm OUTPUT\n") + lines.append("cp " + gadgetfilepath + "P-Gadget3/P-Gadget3 " + filepath + "/P-Gadget3\n") + + if self.submitjob: if self.subtype == "PBS": - f1.write("qsub runscript \n") + lines.append("qsub runscript \n") elif self.subtype == "SLURM": - f1.write("sbatch runscript \n") - - f1.close() + lines.append("sbatch runscript \n") + writers.write(DRIVER_SCRIPT, "".join(lines)) shell.run(["bash", DRIVER_SCRIPT]) print("SUBMITTED JOBS!") @@ -447,8 +443,8 @@ def _vizexpz_button_fired(self): ax.set_xlabel('Snapshot') ax.set_ylabel('Expansion Factor') redall = np.float64(1./expfact - 1) - - if self.alloutputopt == True: + + if self.alloutputopt: self.ExpansionListArr = np.zeros(shape=(len(snapshots),4)) self.ExpansionListArr[:,0] = snapshots self.ExpansionListArr[:,1] = redall @@ -492,7 +488,7 @@ def _vizexpz_button_fired(self): # self.ExpansionListArr[:,2] = io3 # self.ExpansionListArr[:,3] = io4 - + def _expfacti_changed(self): self.redshifti = 1./self.expfacti - 1 @@ -585,12 +581,11 @@ def _checkexistence_button_fired(self): figure = self.main.display figure.clear() ax = figure.add_subplot(111) - + ax = self.main.display.axes[0] ax.set_xticks([]) ax.set_yticks([]) - titlestr = 'HALO BOXTYPE PAD LMIN LMAX NVIR ICS GADGET HALOS' placenormtext(ax,0.03, 0.98,'HALO',10) placenormtext(ax,0.15, 0.98,'BOXTYPE',10) @@ -609,9 +604,8 @@ def _checkexistence_button_fired(self): run = dict(region=boxtypei, padding=paddingi, levelmin=self.lmin, levelmax=self.lmax, overlap=overlapi, nrvir=nrviri, zinit=self.zinit) - foldername = naming.run_dir_name(halo_id=self.haloid, **run) filepath = naming.run_dir(self.gadpath, self.haloid, **run) - + #print filepath #print filepath + '/ics.0' #print filepath + '/outputs/snapdir_064' @@ -619,7 +613,7 @@ def _checkexistence_button_fired(self): try: with open(filepath + '/ics.0'): icfound = '+' - except IOError: + except OSError: icfound = '-' outputs = os.path.join(filepath, 'outputs') lastsnap = int(self.nintervals) - 1 @@ -627,7 +621,7 @@ def _checkexistence_button_fired(self): naming.snapshot_dir(outputs, lastsnap)) else '-' halosfound = '+' if os.path.exists( naming.groups_dir(outputs, lastsnap)) else '-' - + txtdisplace += 0.03 placenormtext(ax,0.02, 0.98 - txtdisplace,str(self.haloid),10) placenormtext(ax,0.14, 0.98 - txtdisplace,str(boxtypei.upper()),10) @@ -638,8 +632,8 @@ def _checkexistence_button_fired(self): placenormtext(ax,0.71, 0.98 - txtdisplace,str(icfound),10) placenormtext(ax,0.81, 0.98 - txtdisplace,str(gadfound),10) placenormtext(ax,0.94, 0.98 - txtdisplace,str(halosfound),10) - - wx.CallAfter(self.main.display.canvas.draw) + + wx.CallAfter(self.main.display.canvas.draw) def __init__(self, main, **kwargs): HasTraits.__init__(self) @@ -741,9 +735,6 @@ def __init__(self, main, **kwargs): self.PBSextraflags = profile['extra_flags'] self._refresh_execute() - #self.PBSstring = \ - #"#!/bin/sh \n#PBS -l nodes=3:ppn=8 \n#PBS -N H190897LX9N1 \n#PBS -m be \n. /opt/torque/etc/openmpi-setup.sh \ncd /bigbang/data/AnnaGroup/caterpillar/halos/H190897/H190897_BE_Z127_P7_LN7_LX9_O4_NV1 \nmpirun -np 24 ./P-Gadget3 ./param.txt 1>OUTPUT 2>ERROR" - diff --git a/src/cme/ui/halos.py b/src/cme/ui/halos.py index 93bec42..f651a70 100644 --- a/src/cme/ui/halos.py +++ b/src/cme/ui/halos.py @@ -1,6 +1,6 @@ +from cme import naming from cme.ui.common import * -from cme import cosmology, naming class HaloFind(HasTraits): @@ -9,14 +9,10 @@ class HaloFind(HasTraits): x_scale = Enum(['linear','log']) y_scale = Enum(['linear','log']) - #haloid = Enum([]) - #haloid = List(Int) haloid = Int() - #Enum(['190897','208737','140666','28221','147419','28188','147273','78411','131988','19910']) snapshot = Range(0,255,255) deltar = Range(0.0,10.0,5.0) - id_select = Int(0) haloidlist = List(Int) datastatus = Str halo_varx = Enum(['id','posX','posY','posZ','corevelx','corevely','corevelz', \ @@ -34,7 +30,7 @@ class HaloFind(HasTraits): 'Xoff','Voff','b_to_a','c_to_a','A[x]','A[y]','A[z]','spin_bullock','T/|U|','npart',\ 'num_cp','numstart','desc','flags','n_core','min_pos_err','min_vel_err','min_bulkvel_err' \ 'hostID','offset','particle_offset']) - + quiver_mode = Enum(['2darrow','2dcircle','2dcross','2ddash','2ddiamond','2dhooked_arrow','2dsquare','2dthick_arrow','2dthick_cross','2dtriangle','2dvertex','arrow','axes','cone','cube','cylinder','point','sphere']) enable_haloglyps = Bool(False) plot_button = Button('Plot') @@ -54,7 +50,6 @@ class HaloFind(HasTraits): lmax = Range(7,14,11,auto_set=False) overlap = Range(1,6,4,auto_set=False) nrvir =Range(1,9,3,auto_set=False) - boxtype = Enum(['ellipsoid','box']) boxtype = Enum(['box','ellipsoid']) parentorzoom = Enum(['zoom','parent']) halo_type = Enum(['all halos','hosts only','subhalos only']) @@ -82,7 +77,7 @@ class HaloFind(HasTraits): enabled_when='fullboxopt=="specific halo" or parentorzoom=="zoom"'), Group(HGroup(Item(name='deltar',label='Box Width [Mpc/h]',springy=True),springy=True) ,enabled_when='fullboxopt=="specific halo" or parentorzoom=="zoom"'), - + Group(Item(name='boxtype',label='Which Volume?',style='custom'),enabled_when='parentorzoom=="zoom"'), HGroup(Group(HGroup(Group(Item(name='nrvir',label='n*rvir(z=0)'), Item(name='lmin',label='Level Min'), @@ -297,7 +292,7 @@ def _plot_button_fired(self): ax.set_yscale(self.y_scale) wx.CallAfter(self.main.display.canvas.draw) - + def _quiver_button_fired(self): self.main.scene.mlab.clf(figure=self.main.scene.mayavi_scene) @@ -312,7 +307,7 @@ def _quiver_button_fired(self): tmphalos = halodata.get_subs() if self.halo_type == 'subs of ID': tmphalos = halodata.get_subhalos_from_halo(self.haloid) - + x = np.array(tmphalos['posX']) y = np.array(tmphalos['posY']) z = np.array(tmphalos['posZ']) @@ -333,7 +328,7 @@ def _quiver_button_fired(self): cond2h = (np.array(halodata.data['posY']) >= yposhost - self.deltar/2.) & (np.array(halodata.data['posY']) <= yposhost + self.deltar/2.) cond3h = (np.array(halodata.data['posZ']) >= zposhost - self.deltar/2.) & (np.array(halodata.data['posZ']) <= zposhost + self.deltar/2.) - condh = cond1h & cond2h & cond3h + condh = cond1h & cond2h & cond3h x = np.array(halodata.data['posX'][condh]) y = np.array(halodata.data['posY'][condh]) @@ -343,11 +338,11 @@ def _quiver_button_fired(self): vz = np.array(halodata.data['pecVZ'][condh]) rvir = np.array(halodata.data['rvir'][condh]) - + vmag = np.sqrt(vx**2+vy**2+vz**2) self.main.scene.mlab.quiver3d(x,y,z,vx,vy,vz,line_width=self.quiver_width,opacity=self.opacity_select,mode=self.quiver_mode,scalars=vmag) - + if self.enable_haloglyps and self.fullboxopt == 'specific halo': self.main.scene.mlab.points3d(x, y, z, rvir/1000) @@ -365,7 +360,6 @@ def _quiver_button_fired(self): def _plotxyzrvir_button_changed(self): self.main.scene.mlab.clf(figure=self.main.scene.mayavi_scene) halodata = RSDataReader.RSDataReader(self.halopath,self.snapshot,digits=2) - boxwidth = self.deltar/2 if self.fullboxopt == 'full box': if self.halo_type == 'all halos': @@ -376,13 +370,13 @@ def _plotxyzrvir_button_changed(self): tmphalos = halodata.get_subs() if self.halo_type == 'subs of ID': tmphalos = halodata.get_subhalos_from_halo(self.haloid) - + x = tmphalos['posX'] y = tmphalos['posY'] z = tmphalos['posZ'] rvir = tmphalos['rvir'] self.main.scene.mlab.points3d(x, y, z, rvir/1000,colormap="copper") - + else: allhalos = halodata.get_hosts() halotmp = allhalos.loc[int(self.haloid)] @@ -397,40 +391,30 @@ def _plotxyzrvir_button_changed(self): condh = cond1h & cond2h & cond3h - condid = (np.array(halodata.data['id']) == self.haloid) - - condnew = condh & condid - x = np.array(halodata.data['posX'][condh]) y = np.array(halodata.data['posY'][condh]) z = np.array(halodata.data['posZ'][condh]) rvir = np.array(halodata.data['rvir'][condh]) - rvirn = np.array(halodata.data['rvir'][condnew]) - mvir = np.array(halodata.data['mvir'][condnew]) - rvmax = np.array(halodata.data['rvmax'][condnew]) - vmax = np.array(halodata.data['vmax'][condnew]) - print(mvir,mvir/cosmology.DEFAULT.h,rvirn,rvmax,vmax,xposhost,yposhost,zposhost) - extent = [xposhost-boxwidth,xposhost+boxwidth,yposhost-boxwidth,yposhost+boxwidth,zposhost-boxwidth,zposhost+boxwidth] self.main.scene.mlab.points3d(x, y, z, rvir/1000,colormap="copper") self.main.scene.mlab.colorbar(orientation='vertical',title='rvir') #self.main.scene.mlab.outline(extent = extent) - #self.main.scene.mlab.axes(extent = extent) - + #self.main.scene.mlab.axes(extent = extent) + self.main.scene.mlab.orientation_axes(xlabel='x-pos',ylabel='x-pos',zlabel='z-pos') self.main.scene.mlab.show() self.main.scene.mlab.axes() self.main.scene.mlab.outline() def _x_scale_changed(self): - if hasattr(self, 'display_points'): + if hasattr(self, 'display_points'): # self.display_points[0].remove() self.main.display.axes[0].set_xscale(self.x_scale) wx.CallAfter(self.main.display.canvas.draw) def _y_scale_changed(self): - if hasattr(self, 'display_points'): + if hasattr(self, 'display_points'): #self.display_points[0].remove() self.main.display.axes[0].set_yscale(self.y_scale) wx.CallAfter(self.main.display.canvas.draw) diff --git a/src/cme/ui/home.py b/src/cme/ui/home.py index 205c0a9..55d2a43 100644 --- a/src/cme/ui/home.py +++ b/src/cme/ui/home.py @@ -1,8 +1,7 @@ """Home tab: pick a cluster profile and confirm the derived paths.""" -from cme.ui.common import * - from cme import config +from cme.ui.common import * class Header(HasTraits): diff --git a/src/cme/ui/ics.py b/src/cme/ui/ics.py index 2da5d15..183d0cd 100644 --- a/src/cme/ui/ics.py +++ b/src/cme/ui/ics.py @@ -1,10 +1,9 @@ -from cme.ui.common import * - # `reWriteIC` post-processes MUSIC's output into the block layout Gadget wants. # It was previously aliased to `re`, shadowing the standard library module. import modules.convertfiles.reWriteIC as rewriteic from cme import cosmology, naming, shell, writers +from cme.ui.common import * from cme.writers import music #: Side length of the Caterpillar parent box, in Mpc/h. @@ -40,9 +39,9 @@ class InitialConditions(HasTraits): cosmologylist = List( editor = CheckListEditor(values = ['WMAP1','WMAP3','WMAP5','WMAP7','WMAP9','PLANCK'],cols = 6) ) boxlength = List(editor = CheckListEditor(values = ['5','10','25','50','75','100','250','500'],cols= 4)) - + zinit = Range(0,127,127) - + padding = List(editor = CheckListEditor(values = ['5','6','7','8','9','10'],cols=6) ) lmin = Enum(['7','8','9','10','11','12','13','14','15']) #lmin = List(editor = CheckListEditor(values = ['7','8','9','10','11','12','13','14','15'],cols=9)) @@ -189,7 +188,7 @@ class InitialConditions(HasTraits): Group(Item(name='lmax',label='Level Max',style='custom'), Item(name='padding',label='Padding',style='custom',springy=False), - Item(name='overlap',label='Overlap',style='custom'),enabled_when='resimbox==True'),show_border=True,label='Iterative Components') + Item(name='overlap',label='Overlap',style='custom'),enabled_when='resimbox==True'),show_border=True,label='Iterative Components') ,label='Setup'), Group(Item(name='fftfine' ,label='fft_fine'), @@ -199,7 +198,7 @@ class InitialConditions(HasTraits): Item(name='smoother',label='smoother',width=-60), Item(name='laplaceorder' ,label='laplace_order',width=-60), Item(name='gradorder' ,label='grad_order',width=-60), - + HGroup(Item(name='baryons' ,label='Baryons'), Item(name='use2LPT' ,label='use_2LPT'), Item(name='useLLA' ,label='use_LLA'), @@ -240,31 +239,31 @@ def run_music(self, confname, workdir, cleanup=False): return result def _parentbox_changed(self): - if self.parentbox == True: + if self.parentbox: self.resimbox = False def _resimbox_changed(self): - if self.resimbox == True: + if self.resimbox: self.parentbox = False def _generate_button_fired(self): if self.haloidselect in self.haloid: - if self.parentbox == True: + if self.parentbox: for cosmi in self.cosmologylist: for boxlengthi in self.boxlength: foldername = cosmi + \ '_L' + str(boxlengthi) + \ '_Z' + str(self.zinit) + \ '_LMIN' + str(self.boxlevel) - + #filepath = self.outpath + 'halos/H' + str(self.haloid) + '/' + foldername - + if not os.path.exists(self.outpath + 'parent/'): os.makedirs(self.outpath + 'parent/') - + if not os.path.exists(self.outpath + 'parent/' + foldername): os.makedirs(self.outpath + 'parent/' + foldername) - + writepath = self.outpath + 'parent/' + foldername confname = os.path.join(writepath, foldername + '.conf') @@ -300,8 +299,8 @@ def _generate_button_fired(self): self.confstatus = 'Generated halo configuration files.' self.run_music(confname, writepath) rewriteic.getBlocks(writepath) - - elif self.parentbox == False: + + elif not self.parentbox: for cosmi in self.cosmologylist: for boxtypei in self.boxtype: for nrviri in self.nrvir: @@ -320,28 +319,28 @@ def _generate_button_fired(self): zinit=self.zinit) foldername = naming.run_dir_name(halo_id=self.haloidselect, **run) filepath = naming.run_dir(self.outpath, self.haloidselect, **run) - + if os.path.exists(filepath): self.confstatus = 'DIR EXIST!' - + elif not os.path.exists(filepath): os.makedirs(filepath) - + #pointfile = dirtest + 'ics/lagr/H' + str(self.haloidselect) + 'NRVIR' + str(int(nrviri)) #pointfile2 = dirtest + 'ics/lagr/H' + str(self.haloidselect) + 'NRVIR' + str(int(nrviri)) - + pointfile = naming.lagr_region_path( os.path.join(self.outpath, 'ics', 'lagr'), self.haloidselect, nrviri) - + #if lmaxi < 13: # pointfile2 = pointfile - writepath = filepath + writepath = filepath #self.outpath + 'halos/' + foldername confname = os.path.join(filepath, foldername + '.conf') #self.outpath + 'halos/' + foldername + '/' + foldername + '.conf' - + self.centx,self.centy,self.centz,self.extx,self.exty,self.extz = getcentext(pointfile + '.head') #print self.parentbox #print self.centx,self.centy,self.centz @@ -382,7 +381,7 @@ def _generate_button_fired(self): )) self.confstatus = 'Generated halo configuration files.' - if self.executemusic == True: + if self.executemusic: self.run_music(confname, writepath, cleanup=True) #cpconvert = "cp ./lib/reWriteIC.py ./lib/convertics.py " + writepath @@ -453,7 +452,7 @@ def _existencebutton_fired(self): try: with open(filename): self.filestatus = 'Lagrangian file exists.' - except IOError: + except OSError: self.filestatus = 'Does not exist.' elif len(self.nrvir) == 0: @@ -476,7 +475,7 @@ def _projopt_changed(self): ytmp = self.ypos xlabel = 'x-pos' ylabel = 'y-pos' - + if self.projopt == 'xz': xbox = self.centx - 0.5*self.extx ybox = self.centz - 0.5*self.extz @@ -486,7 +485,7 @@ def _projopt_changed(self): ytmp = self.zpos xlabel = 'x-pos' ylabel = 'z-pos' - + if self.projopt == 'yz': xbox = self.centy - 0.5*self.exty ybox = self.centz - 0.5*self.extz @@ -496,7 +495,7 @@ def _projopt_changed(self): ytmp = self.zpos xlabel = 'y-pos' ylabel = 'z-pos' - + ax = self.main.display.axes[0] ax.add_patch(patches.Rectangle((xbox,ybox),xext,yext,facecolor='none', edgecolor=self.main.markercolor, @@ -533,11 +532,11 @@ def _makeic_button_fired(self): figure = self.main.display figure.clear() ax = figure.add_subplot(111) - + self.xpos,self.ypos,self.zpos = getlagrxyz(filename) self.centx,self.centy,self.centz,self.extx,self.exty,self.extz = getcentext(filename + '.head') - + if self.projopt == 'xy': xbox = self.centx - 0.5*self.extx ybox = self.centy - 0.5*self.exty @@ -547,7 +546,7 @@ def _makeic_button_fired(self): ytmp = self.ypos xlabel = 'x-pos' ylabel = 'y-pos' - + if self.projopt == 'xz': xbox = self.centx - 0.5*self.extx ybox = self.centz - 0.5*self.extz @@ -557,7 +556,7 @@ def _makeic_button_fired(self): ytmp = self.zpos xlabel = 'x-pos' ylabel = 'z-pos' - + if self.projopt == 'yz': xbox = self.centy - 0.5*self.exty ybox = self.centz - 0.5*self.extz @@ -567,7 +566,7 @@ def _makeic_button_fired(self): ytmp = self.zpos xlabel = 'y-pos' ylabel = 'z-pos' - + ax = self.main.display.axes[0] self.lagrnpart = len(xtmp) self.lagrvol = self.extx*self.exty*self.extz @@ -584,21 +583,21 @@ def _makeic_button_fired(self): ax.set_xlabel(xlabel) ax.set_ylabel(ylabel) wx.CallAfter(self.main.display.canvas.draw) - + else: self.filestatus = 'Not plotting more than one region, select only one nrvir value!' figure = self.main.display figure.clf() ax = figure.add_subplot(111) - except IOError: - try: - with open(self.candidatefiledir + self.candidatefilename): pass - except IOError: - print('CANT FIND CANDIDATE LIST - CHECK DIRECTORIES!') + except OSError: + candidatefile = os.path.join(str(self.candidatefiledir), + str(self.candidatefilename)) + if not os.path.exists(candidatefile): + self.filestatus = 'Cannot find candidate list.' return - if self.writelagrfile == False and len(self.nrvir) != 1: + if not self.writelagrfile and len(self.nrvir) != 1: self.filestatus = 'Cant plot, wont write! I do nothing then!' return @@ -606,9 +605,9 @@ def _makeic_button_fired(self): halodata = RSDataReader.RSDataReader(halopath,63,digits=2) allhalos = halodata.get_hosts() idhalo = int(self.haloidselect) - idcand = getcandidatelist(self.candidatefiledir + self.candidatefilename) + idcand = getcandidatelist(candidatefile) idcand = idcand[:,0] - + matches = np.flatnonzero(idcand == idhalo) if len(matches) == 0: self.filestatus = 'Halo not in candidate list.' @@ -616,14 +615,10 @@ def _makeic_button_fired(self): nhalo = int(matches[0]) rvircand = allhalos.loc[idhalo]['rvir'] - mvircand = allhalos.loc[idhalo]['mvir'] - posXcand = allhalos.loc[idhalo]['posX'] - posYcand = allhalos.loc[idhalo]['posY'] - posZcand = allhalos.loc[idhalo]['posZ'] - + header = rsHD.snapshot_header( self.main.headertab.parentsimpath + PARENT_SNAPSHOT) - + print("------------------------------------------------") print("Rockstar ID inside parent simulation: ",idhalo) print("------------------------------------------------") @@ -631,7 +626,7 @@ def _makeic_button_fired(self): print(" x-pos:",'{:.2f}'.format(float(allhalos.loc[idhalo]['posX'])), " \\ [Mpc/h]") print(" y-pos:",'{:.2f}'.format(float(allhalos.loc[idhalo]['posY'])), " \\ [Mpc/h]") print(" z-pos:",'{:.2f}'.format(float(allhalos.loc[idhalo]['posZ'])), " \\ [Mpc/h]") - print(" virial mass:",'{0:.2e}'.format(float(allhalos.loc[idhalo]['mvir'])/header.hubble),"\\ [Msol]") + print(" virial mass:",'{:.2e}'.format(float(allhalos.loc[idhalo]['mvir'])/header.hubble),"\\ [Msol]") print(" virial radius:",'{:.2f}'.format(float(allhalos.loc[idhalo]['rvir']))," \\ [kpc]") print("------------------------------------------------") @@ -639,7 +634,7 @@ def _makeic_button_fired(self): self.haloposx = '{:.2f}'.format(float(allhalos.loc[idhalo]['posX'])) self.haloposy = '{:.2f}'.format(float(allhalos.loc[idhalo]['posY'])) self.haloposz = '{:.2f}'.format(float(allhalos.loc[idhalo]['posZ'])) - self.halomvir = '{0:.2e}'.format(float(allhalos.loc[idhalo]['mvir'])/header.hubble) + self.halomvir = '{:.2e}'.format(float(allhalos.loc[idhalo]['mvir'])/header.hubble) self.halorvir = '{:.2f}'.format(float(allhalos.loc[idhalo]['rvir'])) # None of these four blocks depends on Nrvir, so read them once @@ -666,7 +661,7 @@ def _makeic_button_fired(self): for Nrvir in self.nrvir: print('Constructing: H' + str(self.haloidselect) + 'NRVIR' + str(int(Nrvir))) Nrvir = float(Nrvir) - Rindex = np.where(R < Nrvir*rvircand/1000) + Rindex = np.where(Nrvir*rvircand/1000 > R) regionIDS = snapIDS[Rindex[0]] mask = np.in1d(snapIDSlagr, regionIDS, assume_unique=True) lagrPos = snapPOSlagr[mask] @@ -685,7 +680,7 @@ def _makeic_button_fired(self): self.extx=2.0*dx*1.12/header.boxsize self.exty=2.0*dy*1.12/header.boxsize self.extz=2.0*dz*1.12/header.boxsize - + if len(self.nrvir) == 1: print("Visualising:", 'HALO' + str(self.haloidselect) + 'NRVIR' + str(int(self.nrvir[0]))) figure = self.main.display @@ -705,7 +700,7 @@ def _makeic_button_fired(self): ytmp = self.ypos xlabel = 'x-pos' ylabel = 'y-pos' - + if self.projopt == 'xz': xbox = self.centx - 0.5*self.extx ybox = self.centz - 0.5*self.extz @@ -715,7 +710,7 @@ def _makeic_button_fired(self): ytmp = self.zpos xlabel = 'x-pos' ylabel = 'z-pos' - + if self.projopt == 'yz': xbox = self.centy - 0.5*self.exty ybox = self.centz - 0.5*self.extz @@ -725,7 +720,7 @@ def _makeic_button_fired(self): ytmp = self.zpos xlabel = 'y-pos' ylabel = 'z-pos' - + ax = self.main.display.axes[0] self.lagrnpart = len(lagrPos[:,1]) self.lagrvol = self.extx*self.exty*self.extz @@ -747,25 +742,18 @@ def _makeic_button_fired(self): self.ypos = lagrPos[:,1] self.zpos = lagrPos[:,2] - if self.writelagrfile == True: + if self.writelagrfile: headerfilename = naming.lagr_header_path( self.lagroutputdir, self.haloidselect, Nrvir) - f1=open(headerfilename,'w') - f1.write('#' + str(self.centx) + '\n') - f1.write('#' + str(self.centy) + '\n') - f1.write('#' + str(self.centz) + '\n') - f1.write('#' + str(self.extx) + '\n') - f1.write('#' + str(self.exty) + '\n') - f1.write('#' + str(self.extz) + '\n') - f1.close() + writers.write(headerfilename, writers.lagrangian_header( + (self.centx, self.centy, self.centz), + (self.extx, self.exty, self.extz))) filename = naming.lagr_region_path( self.lagroutputdir, self.haloidselect, Nrvir) - f2=open(filename,'w') - for iv in range(0,len(lagrPos[:,0])): - f2.write(str(lagrPos[iv,0]/header.boxsize)+' '+str(lagrPos[iv,1]/header.boxsize)+' '+ str(lagrPos[iv,2]/header.boxsize)+'\n') - f2.close() - + writers.write(filename, writers.lagrangian_region( + lagrPos, header.boxsize)) + print("Region constructed!") @@ -816,5 +804,5 @@ def __init__(self, main, **kwargs): try: with open(filename): self.parentconfstatus = "Proceed" - except IOError: + except OSError: self.parentconfstatus = "Cannot locate file." diff --git a/src/cme/ui/install.py b/src/cme/ui/install.py index 1af7549..c431812 100644 --- a/src/cme/ui/install.py +++ b/src/cme/ui/install.py @@ -1,6 +1,6 @@ +from cme import shell from cme.ui.common import * -from cme import shell class InstallFiles(HasTraits): @@ -14,9 +14,9 @@ class InstallFiles(HasTraits): Item('installfftw3_button',show_label=False,springy=True)), HGroup(Item('hdf5dir',show_label=False,springy=True), Item('installhdf5_button',show_label=False,springy=True)), - HGroup(Item('gsldir',show_label=False,springy=True), + HGroup(Item('gsldir',show_label=False,springy=True), Item('installgsl_button',show_label=False,springy=True)), - HGroup(Item('zlibdir',show_label=False,springy=True), + HGroup(Item('zlibdir',show_label=False,springy=True), Item('installzlib_button',show_label=False,springy=True)), HGroup(Item('szipdir',show_label=False,springy=True), Item('installszip_button',show_label=False,springy=True)))) @@ -29,7 +29,7 @@ class InstallFiles(HasTraits): zlibdir = Directory hdf5dir = Directory szipdir = Directory - + installfftw2_button = Button('Install FFTW2') installfftw3_button = Button('Install FFTW3') installhdf5_button = Button('Install HDF5') @@ -129,4 +129,4 @@ def __init__(self, main, **kwargs): self.hdf5dir = self.libpath + '/hdf5' self.gsldir = self.libpath + '/gsl' self.zlibdir = self.libpath + '/zlib' - + diff --git a/src/cme/ui/mergertree.py b/src/cme/ui/mergertree.py index 038c068..697cadc 100644 --- a/src/cme/ui/mergertree.py +++ b/src/cme/ui/mergertree.py @@ -1,6 +1,8 @@ -from cme.ui.common import * import modules.mergertrees.MTCatalogue as MT +from cme.ui.common import * + + class MergerTree(HasTraits): halopath = Directory @@ -34,7 +36,7 @@ class MergerTree(HasTraits): view = View(VGroup(Item(name='halopath',style='readonly'), Item(name='snapshot',label='Snapshot'), - + Group(HGroup(Item(name='haloid',label='Halo ID',style='custom'),springy=True),enabled_when='len(specificid) != 0'), Group(Item(name='specificid',label='Specific ID',width=-80),enabled_when='len(haloid) == 0'), @@ -42,7 +44,7 @@ class MergerTree(HasTraits): Item(name='halo_vary',label='y-axis',springy=True))) ,label='Set Quantities',show_border=True)),enabled_when='use_common == True'), - + HGroup(Group(HGroup(VGroup(Group(Item(name='x_scale',label='x-scale',springy=True), Item(name='y_scale',label='y-scale',springy=True))),label='Set Scale',show_border=True)),enabled_when='use_common == True')), @@ -51,7 +53,7 @@ class MergerTree(HasTraits): Group(Item(name='plot_button',show_label=False,springy=True),enabled_when='len(halo_type) > 0'),enabled_when='len(haloid) >= 1 or specificid != 0'), Group(Item(name='haloidlist',show_label=False,style='readonly',editor=ListEditor(style='readonly',columns=6,rows=4)),label='Halo Sample',show_border=True))) - + def _plot_button_fired(self): if hasattr(self, 'display_points'): figure = self.main.display @@ -73,7 +75,7 @@ def _plot_button_fired(self): tree = cat[0] mainbranch = tree.getMainBranch() self.display_points = ax.plot(mainbranch[self.halo_varx],mainbranch[self.halo_vary],markersize=self.main.markersize,linestyle='-') - + ax.set_xlabel(self.halo_varx) ax.set_ylabel(self.halo_vary) ax.set_xscale(self.x_scale) @@ -82,60 +84,38 @@ def _plot_button_fired(self): wx.CallAfter(self.main.display.canvas.draw) self.jobstatus= "Quantities plotted." - def _halo_varx_changed(self): - self.jobstatus = "Nothing plotted." - if hasattr(self, 'display_points'): - figure = self.main.display - figure.clear() - ax = figure.add_subplot(111) - - ax = self.main.display.axes[0] - wx.CallAfter(self.main.display.canvas.draw) - - def _halo_vary_changed(self): - self.jobstatus = "Nothing plotted." - if hasattr(self, 'display_points'): - figure = self.main.display - figure.clear() - ax = figure.add_subplot(111) - - ax = self.main.display.axes[0] - wx.CallAfter(self.main.display.canvas.draw) - - def _haloid_changed(self): - self.jobstatus = "Nothing plotted." - if hasattr(self, 'display_points'): - figure = self.main.display - figure.clear() - ax = figure.add_subplot(111) - - ax = self.main.display.axes[0] - wx.CallAfter(self.main.display.canvas.draw) + def _invalidate_plot(self): + """Clear the figure when a selection changes, so it cannot go stale. - def _specificid_changed(self): - self.jobstatus = "Nothing plotted." - if hasattr(self, 'display_points'): + Four handlers held byte-identical copies of this, each also fetching an + unused `ax`. + """ + self.jobstatus = "Nothing plotted." + if hasattr(self, 'display_points'): figure = self.main.display figure.clear() - ax = figure.add_subplot(111) + figure.add_subplot(111) + wx.CallAfter(self.main.display.canvas.draw) - ax = self.main.display.axes[0] - wx.CallAfter(self.main.display.canvas.draw) + _halo_varx_changed = _invalidate_plot + _halo_vary_changed = _invalidate_plot + _haloid_changed = _invalidate_plot + _specificid_changed = _invalidate_plot def _x_scale_changed(self): self.jobstatus = "Scale (x) changed." - if hasattr(self, 'display_points'): + if hasattr(self, 'display_points'): # self.display_points[0].remove() self.main.display.axes[0].set_xscale(self.x_scale) wx.CallAfter(self.main.display.canvas.draw) def _y_scale_changed(self): self.jobstatus = "Scale (y) changed." - if hasattr(self, 'display_points'): + if hasattr(self, 'display_points'): #self.display_points[0].remove() self.main.display.axes[0].set_yscale(self.y_scale) wx.CallAfter(self.main.display.canvas.draw) - + def __init__(self, main, **kwargs): HasTraits.__init__(self) self.main = main @@ -145,4 +125,4 @@ def __init__(self, main, **kwargs): self.halopath = self.main.headertab.parentsimpath + 'RockstarData' - + diff --git a/src/cme/writers/__init__.py b/src/cme/writers/__init__.py index e1f5d0b..1f3c43f 100644 --- a/src/cme/writers/__init__.py +++ b/src/cme/writers/__init__.py @@ -20,3 +20,16 @@ def write(path, text): with open(path, "w") as handle: handle.write(text) return path + + +def lagrangian_header(centre, extent): + """The ``.head`` sidecar recording a region's centre and extent. + + Six ``#``-prefixed lines, in the order MUSIC's region reader expects. + """ + return "".join(f"#{value}\n" for value in (*centre, *extent)) + + +def lagrangian_region(positions, boxsize): + """The MUSIC region point file: one box-normalised x y z triple per line.""" + return "".join(f"{x / boxsize} {y / boxsize} {z / boxsize}\n" for x, y, z in positions) diff --git a/src/cme/writers/gadget.py b/src/cme/writers/gadget.py index a569a2c..ef64cad 100644 --- a/src/cme/writers/gadget.py +++ b/src/cme/writers/gadget.py @@ -146,103 +146,163 @@ def param_file( soft = softenings(boxsize, levelmax) if softening is None else softening sections = [ - ("%---- Relevant files", [ - ("InitCondFile", init_cond_file), - ("OutputDir", output_dir), - ("EnergyFile", energy_file), - ("InfoFile", info_file), - ("TimingsFile", timings_file), - ("CpuFile", cpu_file), - ("RestartFile", restart_file), - ("SnapshotFileBase", snapshot_file_base), - ("OutputListFilename", output_list_filename), - ("TimebinFile", timebin_file), - ]), - ("%---- File formats", [ - ("ICFormat", ic_format), - ("SnapFormat", snap_format), - ]), - ("%---- CPU-time limits", [ - ("TimeLimitCPU", time_limit_cpu), - ("CpuTimeBetRestartFile", cpu_time_bet_restart_file), - ("ResubmitOn", resubmit_on), - ("ResubmitCommand", resubmit_command), - ]), - ("%----- Memory allocation", [ - ("MaxMemSize", max_mem_size), - ("PartAllocFactor", part_alloc_factor), - ("BufferSize", buffer_size), - ]), - ("%---- Characteristics of run", [ - ("TimeBegin", time_begin), - ("TimeMax", time_max), - ]), - ("%---- Basic code options that set the type of simulation", [ - ("ComovingIntegrationOn", comoving_integration_on), - ("PeriodicBoundariesOn", periodic_boundaries_on), - ("CoolingOn", cooling_on), - ("StarformationOn", star_formation_on), - ]), - ("%---- Cosmological parameters", [ - ("Omega0", cosmo.omega_m), - ("OmegaLambda", cosmo.omega_l), - ("OmegaBaryon", omega_baryon), - ("HubbleParam", cosmo.h), - ("BoxSize", boxsize), - ]), - ("%---- Output frequency", [ - ("OutputListOn", output_list_on), - ("TimeBetSnapshot", time_bet_snapshot), - ("TimeOfFirstSnapshot", time_of_first_snapshot), - ("TimeBetStatistics", time_bet_statistics), - ("NumFilesPerSnapshot", num_files_per_snapshot), - ("NumFilesWrittenInParallel", num_files_written_in_parallel), - ]), - ("%---- Accuracy of time integration", [ - ("TypeOfTimestepCriterion", type_of_timestep_criterion), - ("ErrTolIntAccuracy", err_tol_int_accuracy), - ("CourantFac", courant_fac), - ("MaxRMSDisplacementFac", max_rms_displacement_fac), - ("MaxSizeTimestep", max_size_timestep), - ("MinSizeTimestep", min_size_timestep), - ]), - ("%---- Gas temperatures", [ - ("InitGasTemp", init_gas_temp), - ("MinGasTemp", min_gas_temp), - ]), - ("%---- Tree algorithm, force accuracy, domain update frequency", [ - ("TypeOfOpeningCriterion", type_of_opening_criterion), - ("ErrTolTheta", err_tol_theta), - ("ErrTolForceAcc", err_tol_force_acc), - ("TreeDomainUpdateFrequency", tree_domain_update_frequency), - ]), - ("%---- Initial density estimate", [ - ("DesNumNgb", des_num_ngb), - ("MaxNumNgbDeviation", max_num_ngb_deviation), - ]), - ("%---- System of units", [ - ("UnitLength_in_cm", unit_length_in_cm), - ("UnitMass_in_g", unit_mass_in_g), - ("UnitVelocity_in_cm_per_s", unit_velocity_in_cm_per_s), - ("GravityConstantInternal", gravity_constant_internal), - ]), - ("%---- Gravitational softening lengths", [ - ("MinGasHsmlFractional", min_gas_hsml_fractional), - ]), - (None, [(f"Softening{t}", soft[t]) - for t in ("Gas", "Halo", "Disk", "Bulge", "Stars", "Bndry")]), - (None, [(f"Softening{t}MaxPhys", soft[t]) - for t in ("Gas", "Halo", "Disk", "Bulge", "Stars", "Bndry")]), - ("%---- non-common", [ - ("ArtBulkViscConst", art_bulk_visc_const), - ]), + ( + "%---- Relevant files", + [ + ("InitCondFile", init_cond_file), + ("OutputDir", output_dir), + ("EnergyFile", energy_file), + ("InfoFile", info_file), + ("TimingsFile", timings_file), + ("CpuFile", cpu_file), + ("RestartFile", restart_file), + ("SnapshotFileBase", snapshot_file_base), + ("OutputListFilename", output_list_filename), + ("TimebinFile", timebin_file), + ], + ), + ( + "%---- File formats", + [ + ("ICFormat", ic_format), + ("SnapFormat", snap_format), + ], + ), + ( + "%---- CPU-time limits", + [ + ("TimeLimitCPU", time_limit_cpu), + ("CpuTimeBetRestartFile", cpu_time_bet_restart_file), + ("ResubmitOn", resubmit_on), + ("ResubmitCommand", resubmit_command), + ], + ), + ( + "%----- Memory allocation", + [ + ("MaxMemSize", max_mem_size), + ("PartAllocFactor", part_alloc_factor), + ("BufferSize", buffer_size), + ], + ), + ( + "%---- Characteristics of run", + [ + ("TimeBegin", time_begin), + ("TimeMax", time_max), + ], + ), + ( + "%---- Basic code options that set the type of simulation", + [ + ("ComovingIntegrationOn", comoving_integration_on), + ("PeriodicBoundariesOn", periodic_boundaries_on), + ("CoolingOn", cooling_on), + ("StarformationOn", star_formation_on), + ], + ), + ( + "%---- Cosmological parameters", + [ + ("Omega0", cosmo.omega_m), + ("OmegaLambda", cosmo.omega_l), + ("OmegaBaryon", omega_baryon), + ("HubbleParam", cosmo.h), + ("BoxSize", boxsize), + ], + ), + ( + "%---- Output frequency", + [ + ("OutputListOn", output_list_on), + ("TimeBetSnapshot", time_bet_snapshot), + ("TimeOfFirstSnapshot", time_of_first_snapshot), + ("TimeBetStatistics", time_bet_statistics), + ("NumFilesPerSnapshot", num_files_per_snapshot), + ("NumFilesWrittenInParallel", num_files_written_in_parallel), + ], + ), + ( + "%---- Accuracy of time integration", + [ + ("TypeOfTimestepCriterion", type_of_timestep_criterion), + ("ErrTolIntAccuracy", err_tol_int_accuracy), + ("CourantFac", courant_fac), + ("MaxRMSDisplacementFac", max_rms_displacement_fac), + ("MaxSizeTimestep", max_size_timestep), + ("MinSizeTimestep", min_size_timestep), + ], + ), + ( + "%---- Gas temperatures", + [ + ("InitGasTemp", init_gas_temp), + ("MinGasTemp", min_gas_temp), + ], + ), + ( + "%---- Tree algorithm, force accuracy, domain update frequency", + [ + ("TypeOfOpeningCriterion", type_of_opening_criterion), + ("ErrTolTheta", err_tol_theta), + ("ErrTolForceAcc", err_tol_force_acc), + ("TreeDomainUpdateFrequency", tree_domain_update_frequency), + ], + ), + ( + "%---- Initial density estimate", + [ + ("DesNumNgb", des_num_ngb), + ("MaxNumNgbDeviation", max_num_ngb_deviation), + ], + ), + ( + "%---- System of units", + [ + ("UnitLength_in_cm", unit_length_in_cm), + ("UnitMass_in_g", unit_mass_in_g), + ("UnitVelocity_in_cm_per_s", unit_velocity_in_cm_per_s), + ("GravityConstantInternal", gravity_constant_internal), + ], + ), + ( + "%---- Gravitational softening lengths", + [ + ("MinGasHsmlFractional", min_gas_hsml_fractional), + ], + ), + ( + None, + [ + (f"Softening{t}", soft[t]) + for t in ("Gas", "Halo", "Disk", "Bulge", "Stars", "Bndry") + ], + ), + ( + None, + [ + (f"Softening{t}MaxPhys", soft[t]) + for t in ("Gas", "Halo", "Disk", "Bulge", "Stars", "Bndry") + ], + ), + ( + "%---- non-common", + [ + ("ArtBulkViscConst", art_bulk_visc_const), + ], + ), ] if include_subfind: - sections.append((None, [ - ("ErrTolThetaSubfind", err_tol_theta_subfind), - ("DesLinkNgb", des_link_ngb), - ])) + sections.append( + ( + None, + [ + ("ErrTolThetaSubfind", err_tol_theta_subfind), + ("DesLinkNgb", des_link_ngb), + ], + ) + ) out = [] for heading, pairs in sections: @@ -257,56 +317,90 @@ def param_file( #: emitted commented out, which is how Gadget's Config.sh expresses "off" while #: keeping the option discoverable. _CONFIG_SECTIONS = [ - ("Basic operation mode of code", [ - ("PERIODIC", True, None), - ("COOLING", False, None), - ("SFR", False, None), - ("SINKS", False, None), - ("UNEQUALSOFTENINGS", True, None), - ("NUM_THREADS=4", False, - "Now OpenMP works the same, so don't compile with OpenMP *and* PTHREADS !"), - ]), - ("Kernel Options", [ - ("QUINTIC_KERNEL", False, - "Implementation of the Morris 1996 quintic spline kernel, requires (3/2)^3 more neighbours !"), - ("TWODIMS", False, "Switch for 2D test problems"), - ("ONEDIM", False, "Switch for 1D test problems"), - ]), - ("Multi-Domain and Top-Level Tree options", [ - ("MULTIPLEDOMAINS=8", True, None), - ("TOPNODEFACTOR=3.0", False, None), - ("KD_HMAX_ESTIMATE", False, - "Alternative way to update HMAX within Tree nodes"), - ]), - ("Things that are always recommended", [ - ("PEANOHILBERT", True, None), - ("WALLCLOCK", True, None), - ("MYSORT", True, None), - ("AUTO_SWAP_ENDIAN_READIC", False, - "Enables automatic ENDIAN swapping for reading ICs"), - ("WRITE_KEY_FILES", False, "Enables writing key index files"), - ("WRITE_INFO_BLOCK", False, "Enables writing the INFO block"), - ("PERMUTATAION_OPTIMIZATION", False, None), - ("PROCESS_TIMES_OF_OUTPUTLIST", False, - "Chooses the outputtime closest to any global step"), - ("SYNCRONIZ_OUTPUT", False, "Writes output only at global time steps"), - ]), - ("Single/Double Precision", [ - ("DOUBLEPRECISION", True, None), - ("DOUBLEPRECISION_FFTW", True, None), - ("OUTPUT_IN_DOUBLEPRECISION", False, - "snapshot files will be written in double precision"), - ("INPUT_IN_DOUBLEPRECISION", False, None), - ]), - ("Invariance Test", [ - ("INVARIANCETEST", False, None), - ("INVARIANCETEST_SIZE1=2", False, None), - ("INVARIANCETEST_SIZE2=6", False, None), - ("FLTROUNDOFFREDUCTION", False, - "enables (expensive!) `double-double' round-off reduction in particle sums"), - ("SOFTDOUBLEDOUBLE", False, - "needs to be set if a C++ software implementation of 128bit double-double precision should be used"), - ]), + ( + "Basic operation mode of code", + [ + ("PERIODIC", True, None), + ("COOLING", False, None), + ("SFR", False, None), + ("SINKS", False, None), + ("UNEQUALSOFTENINGS", True, None), + ( + "NUM_THREADS=4", + False, + "Now OpenMP works the same, so don't compile with OpenMP *and* PTHREADS !", + ), + ], + ), + ( + "Kernel Options", + [ + ( + "QUINTIC_KERNEL", + False, + "Implementation of the Morris 1996 quintic spline kernel, requires (3/2)^3 more neighbours !", + ), + ("TWODIMS", False, "Switch for 2D test problems"), + ("ONEDIM", False, "Switch for 1D test problems"), + ], + ), + ( + "Multi-Domain and Top-Level Tree options", + [ + ("MULTIPLEDOMAINS=8", True, None), + ("TOPNODEFACTOR=3.0", False, None), + ("KD_HMAX_ESTIMATE", False, "Alternative way to update HMAX within Tree nodes"), + ], + ), + ( + "Things that are always recommended", + [ + ("PEANOHILBERT", True, None), + ("WALLCLOCK", True, None), + ("MYSORT", True, None), + ("AUTO_SWAP_ENDIAN_READIC", False, "Enables automatic ENDIAN swapping for reading ICs"), + ("WRITE_KEY_FILES", False, "Enables writing key index files"), + ("WRITE_INFO_BLOCK", False, "Enables writing the INFO block"), + ("PERMUTATAION_OPTIMIZATION", False, None), + ( + "PROCESS_TIMES_OF_OUTPUTLIST", + False, + "Chooses the outputtime closest to any global step", + ), + ("SYNCRONIZ_OUTPUT", False, "Writes output only at global time steps"), + ], + ), + ( + "Single/Double Precision", + [ + ("DOUBLEPRECISION", True, None), + ("DOUBLEPRECISION_FFTW", True, None), + ( + "OUTPUT_IN_DOUBLEPRECISION", + False, + "snapshot files will be written in double precision", + ), + ("INPUT_IN_DOUBLEPRECISION", False, None), + ], + ), + ( + "Invariance Test", + [ + ("INVARIANCETEST", False, None), + ("INVARIANCETEST_SIZE1=2", False, None), + ("INVARIANCETEST_SIZE2=6", False, None), + ( + "FLTROUNDOFFREDUCTION", + False, + "enables (expensive!) `double-double' round-off reduction in particle sums", + ), + ( + "SOFTDOUBLEDOUBLE", + False, + "needs to be set if a C++ software implementation of 128bit double-double precision should be used", + ), + ], + ), ] #: SUBFIND-related options, emitted enabled or commented out together. @@ -317,38 +411,61 @@ def param_file( _SUBFIND_TRAILER = [ ("FOF_GROUP_MIN_LEN=32", False, "default is 32"), - ("DENSITY_SPLIT_BY_TYPE=1+2+16+32", False, - "2^type for whch the densities should be calculated seperately"), - ("MAX_NGB_CHECK=3", False, - "Max numbers of neighbours for sattlepoint detection (default = 2)"), - ("SAVE_MASS_TAB", False, - "Saves the an additional array with the masses of the different components"), - ("SUBFINDSAVE_PARTICLELISTS", False, - "Saves also phase-space and type variables parallel to IDs"), - ("SO_VEL_DISPERSIONS", False, - "computes velocity dispersions for as part of FOF SO-properties"), + ( + "DENSITY_SPLIT_BY_TYPE=1+2+16+32", + False, + "2^type for whch the densities should be calculated seperately", + ), + ("MAX_NGB_CHECK=3", False, "Max numbers of neighbours for sattlepoint detection (default = 2)"), + ( + "SAVE_MASS_TAB", + False, + "Saves the an additional array with the masses of the different components", + ), + ( + "SUBFINDSAVE_PARTICLELISTS", + False, + "Saves also phase-space and type variables parallel to IDs", + ), + ("SO_VEL_DISPERSIONS", False, "computes velocity dispersions for as part of FOF SO-properties"), ("ORDER_SNAPSHOTS_BY_ID", False, None), - ("SAVE_HSML_IN_IC_ORDER", False, - "will store the hsml-values in the order of the particles in the IC file"), + ( + "SAVE_HSML_IN_IC_ORDER", + False, + "will store the hsml-values in the order of the particles in the IC file", + ), ("ONLY_PRODUCE_HSML_FILES", False, "only carries out density estimate"), - ("KEEP_HSML_AS_GUESS", False, - "keep using hsml for gas particles in subfind_density"), + ("KEEP_HSML_AS_GUESS", False, "keep using hsml for gas particles in subfind_density"), ("LINKLENGTH=0.16", False, "Linkinglength for FoF (default=0.2)"), ("NO_GAS_CLOUDS", False, "Do not accept pure gaseous substructures"), ("WRITE_SUB_IN_SNAP_FORMAT", False, "Save subfind results in snap format"), - ("LT_ADD_GAL_TO_SUB=12", False, - "Adds optical luminosities in 6 bands to subhalos"), - ("DUSTATT=11", False, - "Includes dust attenuation into the luminosity calculation (using 11 radial bins)"), - ("OBSERVER_FRAME", False, - "If defined, use CB07 Observer Frame Luminosities, otherwise CB07 Rest Frame Luminosities"), + ("LT_ADD_GAL_TO_SUB=12", False, "Adds optical luminosities in 6 bands to subhalos"), + ( + "DUSTATT=11", + False, + "Includes dust attenuation into the luminosity calculation (using 11 radial bins)", + ), + ( + "OBSERVER_FRAME", + False, + "If defined, use CB07 Observer Frame Luminosities, otherwise CB07 Rest Frame Luminosities", + ), ("SO_BAR_INFO", False, "Adds temperature, Lx, bfrac, etc to Groups"), - ("FSUBFINDCOUNT_BIG_HALOS=1e4", False, - "Adds extra blocks for Halos with M_TopHat > SUBFIND_COUNT_BIG_HALOS"), - ("KD_CHOOSE_PSUBFIND_LIMIT", False, - "Increases the limit for the parallel subfind to the maximum possible"), - ("KD_ALTERNATIVE_GROUP_SORT", False, - "Alternative way to sort the Groups/SubGroupe before writing"), + ( + "FSUBFINDCOUNT_BIG_HALOS=1e4", + False, + "Adds extra blocks for Halos with M_TopHat > SUBFIND_COUNT_BIG_HALOS", + ), + ( + "KD_CHOOSE_PSUBFIND_LIMIT", + False, + "Increases the limit for the parallel subfind to the maximum possible", + ), + ( + "KD_ALTERNATIVE_GROUP_SORT", + False, + "Alternative way to sort the Groups/SubGroupe before writing", + ), ("KD_CHOOSE_LINKING_LENGTH", False, "Special way to estimate the linking length"), ("SUBFINDREAD_FOF", False, None), ("SUBFINDCOLLECTIVE_STAGE1", False, None), @@ -356,10 +473,16 @@ def param_file( ("SUBFINDALTERNATIVE_COLLECTIVE", False, None), ("SUBFINDRESHUFFLE_CATALOGUE", False, None), ("SUBFINDRESHUFFLE_CATALOGUE_WITH_VORONOI", False, None), - ("SUBFINDRESHUFFLE_AND_POTENTIAL", False, - "needs -DSUBFIND_RESHUFFLE_CATALOGUE and COMPUTE_POTENTIAL_ENERGY"), - ("SUBFINDDENSITY_AND_POTENTIAL", False, - "only calculated density and potential and write them into snapshot"), + ( + "SUBFINDRESHUFFLE_AND_POTENTIAL", + False, + "needs -DSUBFIND_RESHUFFLE_CATALOGUE and COMPUTE_POTENTIAL_ENERGY", + ), + ( + "SUBFINDDENSITY_AND_POTENTIAL", + False, + "only calculated density and potential and write them into snapshot", + ), ] @@ -414,14 +537,16 @@ def emit(section): emit(options) out.append(_heading("TreePM Options")) - emit([ - (f"PMGRID={int(pmgrid)}", True, None), - (f"GRIDBOOST={gridboost}", True, None), - ("#ASMTH=1.25", False, None), - ("RCUT=5.25", False, None), - (f"PLACEHIGHRESREGION={placehighresregion}", True, None), - (f"ENLARGEREGION={enlargeregion}", True, None), - ]) + emit( + [ + (f"PMGRID={int(pmgrid)}", True, None), + (f"GRIDBOOST={gridboost}", True, None), + ("#ASMTH=1.25", False, None), + ("RCUT=5.25", False, None), + (f"PLACEHIGHRESREGION={placehighresregion}", True, None), + (f"ENLARGEREGION={enlargeregion}", True, None), + ] + ) for heading, options in _CONFIG_SECTIONS[2:]: out.append(_heading(heading)) @@ -430,11 +555,13 @@ def emit(section): out.append(_heading("On the fly FOF groupfinder")) for token, comment in _SUBFIND_SECTION: out.append(_option(token, include_subfind, comment)) - out.append(_option( - f"FOF_SECONDARY_LINK_TYPES={fof_secondary_link_types}", - include_subfind, - "2^type for the types linked to nearest primaries", - )) + out.append( + _option( + f"FOF_SECONDARY_LINK_TYPES={fof_secondary_link_types}", + include_subfind, + "2^type for the types linked to nearest primaries", + ) + ) out.append(_option("FOF_GROUP_MIN_LEN=32", False, "default is 32")) out.append(_option("SUBFIND", include_subfind, None)) for token, enabled, comment in _SUBFIND_TRAILER[1:]: diff --git a/src/cme/writers/music.py b/src/cme/writers/music.py index b58cd88..d6f8c8f 100644 --- a/src/cme/writers/music.py +++ b/src/cme/writers/music.py @@ -186,8 +186,13 @@ def resim_conf( _section("random", [(f"seed[{level}]", seed) for level, seed in seeds]), _output_section(out_format, ic_filename, num_files), _poisson_section( - fft_fine, accuracy, pre_smooth, post_smooth, - smoother, laplace_order, grad_order, + fft_fine, + accuracy, + pre_smooth, + post_smooth, + smoother, + laplace_order, + grad_order, ), ] ) @@ -251,8 +256,13 @@ def parent_conf( _section("random", [(f"seed[{int(level)}]", int(seed))]), _output_section(out_format, ic_filename, num_files), _poisson_section( - fft_fine, accuracy, pre_smooth, post_smooth, - smoother, laplace_order, grad_order, + fft_fine, + accuracy, + pre_smooth, + post_smooth, + smoother, + laplace_order, + grad_order, ), ] ) diff --git a/src/cme/writers/submit.py b/src/cme/writers/submit.py index eb7415f..59375e4 100644 --- a/src/cme/writers/submit.py +++ b/src/cme/writers/submit.py @@ -51,9 +51,7 @@ def mpirun_command( else: args = "" - return ( - f"mpirun -np {int(ranks)} ./{binary} ./{param}{args} 1>{out} 2>{err}" - ) + return f"mpirun -np {int(ranks)} ./{binary} ./{param}{args} 1>{out} 2>{err}" def slurm_script( diff --git a/tests/test_config.py b/tests/test_config.py index 2ea17c9..e2ce252 100644 --- a/tests/test_config.py +++ b/tests/test_config.py @@ -8,9 +8,7 @@ os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "clusters.toml" ) -pytestmark = pytest.mark.skipif( - config.tomllib is None, reason="no TOML parser available" -) +pytestmark = pytest.mark.skipif(config.tomllib is None, reason="no TOML parser available") @pytest.fixture @@ -21,7 +19,12 @@ def cfg(): class TestLoad: def test_reads_the_repository_profiles(self, cfg): assert set(cfg["clusters"]) >= { - "local", "bigbang", "odyssey", "spacebase", "antares", "barrine" + "local", + "bigbang", + "odyssey", + "spacebase", + "antares", + "barrine", } def test_falls_back_to_a_local_profile_when_no_file_is_found(self, monkeypatch): @@ -55,9 +58,7 @@ class TestDetect: ("barrine", "barrine"), ], ) - def test_recognises_the_hostnames_the_if_chain_handled( - self, cfg, hostname, expected - ): + def test_recognises_the_hostnames_the_if_chain_handled(self, cfg, hostname, expected): assert config.detect(cfg, hostname=hostname) == expected def test_unknown_hostname_falls_back_to_the_default(self, cfg): diff --git a/tests/test_cosmology.py b/tests/test_cosmology.py index 490231b..0ff202d 100644 --- a/tests/test_cosmology.py +++ b/tests/test_cosmology.py @@ -35,9 +35,7 @@ def test_matches_the_hardcoded_gadget_defaults(self): def test_omega_b_is_consistent_with_planck_2013_omega_b_h_squared(self): # Planck 2013 XVI reports Omega_b h^2 = 0.02205. - assert cosmology.PLANCK.omega_b == pytest.approx( - 0.02205 / cosmology.PLANCK.h**2, abs=1e-3 - ) + assert cosmology.PLANCK.omega_b == pytest.approx(0.02205 / cosmology.PLANCK.h**2, abs=1e-3) class TestInternalConsistency: @@ -61,7 +59,7 @@ def test_every_set_cites_a_reference(self, name): class TestUnitConvention: def test_H0_is_in_km_per_s_per_mpc(self): - assert cosmology.PLANCK.H0 == pytest.approx(67.11) + assert pytest.approx(67.11) == cosmology.PLANCK.H0 def test_as_tuple_returns_H0_not_h(self): # The Gadget tab writes `HubbleParam` as this value / 100, so the tuple @@ -71,9 +69,7 @@ def test_as_tuple_returns_H0_not_h(self): assert hubble / 100 == pytest.approx(cosmology.PLANCK.h) def test_cosmoconstant_shim_matches_the_old_six_tuple_order(self): - omega_m, omega_l, omega_b, hubble, sigma_8, n_spec = ( - cosmology.cosmoconstant("PLANCK") - ) + omega_m, omega_l, omega_b, hubble, sigma_8, n_spec = cosmology.cosmoconstant("PLANCK") assert (omega_m, omega_l, omega_b) == (0.3175, 0.6825, 0.0490) assert (hubble, sigma_8, n_spec) == pytest.approx((67.11, 0.8344, 0.9624)) diff --git a/tests/test_naming.py b/tests/test_naming.py index b751f37..1bec385 100644 --- a/tests/test_naming.py +++ b/tests/test_naming.py @@ -36,8 +36,13 @@ def test_matches_the_established_convention(self): def test_box_region_uses_b(self): name = naming.run_dir_name( - halo_id=1930, region="box", padding=8, levelmin=7, - levelmax=12, overlap=4, nrvir=3, + halo_id=1930, + region="box", + padding=8, + levelmin=7, + levelmax=12, + overlap=4, + nrvir=3, ) assert name == "H1930_BB_Z127_P8_LN7_LX12_O4_NV3" @@ -45,12 +50,22 @@ def test_accepts_strings_as_well_as_ints(self): # The ICs and Gadget tabs supply these as strings from a # CheckListEditor; the contamination and halo tabs supply ints. from_strings = naming.run_dir_name( - halo_id="190897", region="ellipsoid", padding="7", levelmin="7", - levelmax="11", overlap="4", nrvir="4", + halo_id="190897", + region="ellipsoid", + padding="7", + levelmin="7", + levelmax="11", + overlap="4", + nrvir="4", ) from_ints = naming.run_dir_name( - halo_id=190897, region="ellipsoid", padding=7, levelmin=7, - levelmax=11, overlap=4, nrvir=4, + halo_id=190897, + region="ellipsoid", + padding=7, + levelmin=7, + levelmax=11, + overlap=4, + nrvir=4, ) assert from_strings == from_ints @@ -59,20 +74,37 @@ def test_zinit_defaults_to_127_but_is_honoured(self): # non-default zinit silently pointed them at a directory that the ICs # tab had never created. assert "_Z127_" in naming.run_dir_name( - halo_id=1, region="box", padding=7, levelmin=7, - levelmax=11, overlap=4, nrvir=1, + halo_id=1, + region="box", + padding=7, + levelmin=7, + levelmax=11, + overlap=4, + nrvir=1, ) assert "_Z63_" in naming.run_dir_name( - halo_id=1, region="box", padding=7, levelmin=7, - levelmax=11, overlap=4, nrvir=1, zinit=63, + halo_id=1, + region="box", + padding=7, + levelmin=7, + levelmax=11, + overlap=4, + nrvir=1, + zinit=63, ) class TestParseRunDirName: def test_round_trips(self): kwargs = dict( - halo_id=241932, region="box", padding=9, levelmin=8, - levelmax=13, overlap=5, nrvir=7, zinit=63, + halo_id=241932, + region="box", + padding=9, + levelmin=8, + levelmax=13, + overlap=5, + nrvir=7, + zinit=63, ) parsed = naming.parse_run_dir_name(naming.run_dir_name(**kwargs)) assert parsed == kwargs @@ -105,8 +137,12 @@ def test_regression_three_digit_snapshots(self): class TestPaths: RUN = dict( - region="ellipsoid", padding=7, levelmin=7, - levelmax=11, overlap=4, nrvir=4, + region="ellipsoid", + padding=7, + levelmin=7, + levelmax=11, + overlap=4, + nrvir=4, ) def test_run_dir(self): @@ -118,9 +154,7 @@ def test_outputs_and_children(self): outputs = naming.outputs_dir("/data", 190897, **self.RUN) assert outputs.endswith("/H190897_BE_Z127_P7_LN7_LX11_O4_NV4/outputs") assert naming.groups_dir(outputs, 63).endswith("/outputs/groups_063") - assert naming.snapshot_base(outputs, 63).endswith( - "/outputs/snapdir_063/snap_063" - ) + assert naming.snapshot_base(outputs, 63).endswith("/outputs/snapdir_063/snap_063") def test_rockstar_dir(self): assert naming.rockstar_dir("/data", 190897, **self.RUN).endswith( @@ -136,9 +170,7 @@ def test_trailing_slash_on_base_is_normalised(self): ) def test_lagrangian_paths(self): - assert naming.lagr_region_path("/d/ics/lagr", 190897, 4) == ( - "/d/ics/lagr/H190897NRVIR4" - ) + assert naming.lagr_region_path("/d/ics/lagr", 190897, 4) == ("/d/ics/lagr/H190897NRVIR4") assert naming.lagr_header_path("/d/ics/lagr", 190897, 4) == ( "/d/ics/lagr/H190897NRVIR4.head" ) diff --git a/tests/test_science_candidates.py b/tests/test_science_candidates.py index 93991a0..a064415 100644 --- a/tests/test_science_candidates.py +++ b/tests/test_science_candidates.py @@ -29,19 +29,23 @@ def test_is_relative_reflects_which_was_given(self): class TestMassRange: def test_selects_only_haloes_inside_the_range(self): - cat = catalogue([ - (1, 1e11, 100.0, 10.0, 10.0, 10.0), # too light - (2, MW, 200.0, 30.0, 30.0, 30.0), # in range - (3, 1e14, 900.0, 50.0, 50.0, 50.0), # too heavy - ]) + cat = catalogue( + [ + (1, 1e11, 100.0, 10.0, 10.0, 10.0), # too light + (2, MW, 200.0, 30.0, 30.0, 30.0), # in range + (3, 1e14, 900.0, 50.0, 50.0, 50.0), # too heavy + ] + ) out = candidates.select(cat, RANGE, [], hubble=1.0) assert list(out["id"]) == [2] def test_range_is_exclusive_at_both_ends(self): - cat = catalogue([ - (1, 7e11, 100.0, 10.0, 10.0, 10.0), - (2, 7e12, 100.0, 30.0, 30.0, 30.0), - ]) + cat = catalogue( + [ + (1, 7e11, 100.0, 10.0, 10.0, 10.0), + (2, 7e12, 100.0, 30.0, 30.0, 30.0), + ] + ) assert len(candidates.select(cat, RANGE, [], hubble=1.0)) == 0 def test_returns_an_empty_array_not_a_zero_row(self): @@ -64,10 +68,12 @@ def test_masses_are_converted_before_the_range_test(self): def test_distances_are_converted_before_the_zone_test(self): # Separation is 2 Mpc/h. At h=1.0 that is 2 Mpc and clears a 1.5 Mpc # zone; at h=0.5 it is 4 Mpc and clears it by more. - cat = catalogue([ - (1, MW, 200.0, 10.0, 10.0, 10.0), - (2, 1e14, 900.0, 12.0, 10.0, 10.0), - ]) + cat = catalogue( + [ + (1, MW, 200.0, 10.0, 10.0, 10.0), + (2, 1e14, 900.0, 12.0, 10.0, 10.0), + ] + ) zone = [ExclusionZone(distance=3.0, mass=1e13)] assert len(candidates.select(cat, RANGE, zone, hubble=1.0)) == 0 assert len(candidates.select(cat, RANGE, zone, hubble=0.5)) == 1 @@ -75,36 +81,44 @@ def test_distances_are_converted_before_the_zone_test(self): class TestAbsoluteExclusionZones: def test_rejects_a_candidate_with_a_massive_neighbour_inside_the_zone(self): - cat = catalogue([ - (1, MW, 200.0, 10.0, 10.0, 10.0), - (2, 1e14, 900.0, 12.0, 10.0, 10.0), # 2 Mpc away - ]) + cat = catalogue( + [ + (1, MW, 200.0, 10.0, 10.0, 10.0), + (2, 1e14, 900.0, 12.0, 10.0, 10.0), # 2 Mpc away + ] + ) zones = [ExclusionZone(distance=4.0, mass=7e13)] assert len(candidates.select(cat, RANGE, zones, hubble=1.0)) == 0 def test_accepts_when_the_neighbour_is_beyond_the_zone(self): - cat = catalogue([ - (1, MW, 200.0, 10.0, 10.0, 10.0), - (2, 1e14, 900.0, 20.0, 10.0, 10.0), # 10 Mpc away - ]) + cat = catalogue( + [ + (1, MW, 200.0, 10.0, 10.0, 10.0), + (2, 1e14, 900.0, 20.0, 10.0, 10.0), # 10 Mpc away + ] + ) zones = [ExclusionZone(distance=4.0, mass=7e13)] assert list(candidates.select(cat, RANGE, zones, hubble=1.0)["id"]) == [1] def test_accepts_when_the_neighbour_is_below_the_mass_threshold(self): - cat = catalogue([ - (1, MW, 200.0, 10.0, 10.0, 10.0), - # Close, but far below the zone's mass -- and below the base sample - # range too, so it is not itself a candidate. - (2, 1e11, 90.0, 11.0, 10.0, 10.0), - ]) + cat = catalogue( + [ + (1, MW, 200.0, 10.0, 10.0, 10.0), + # Close, but far below the zone's mass -- and below the base sample + # range too, so it is not itself a candidate. + (2, 1e11, 90.0, 11.0, 10.0, 10.0), + ] + ) zones = [ExclusionZone(distance=4.0, mass=7e13)] assert list(candidates.select(cat, RANGE, zones, hubble=1.0)["id"]) == [1] def test_all_zones_must_pass(self): - cat = catalogue([ - (1, MW, 200.0, 10.0, 10.0, 10.0), - (2, 8e12, 300.0, 12.0, 10.0, 10.0), # violates the near zone only - ]) + cat = catalogue( + [ + (1, MW, 200.0, 10.0, 10.0, 10.0), + (2, 8e12, 300.0, 12.0, 10.0, 10.0), # violates the near zone only + ] + ) near = ExclusionZone(distance=3.0, mass=7e12) far = ExclusionZone(distance=4.0, mass=7e13) assert len(candidates.select(cat, RANGE, [far], hubble=1.0)) == 1 @@ -112,14 +126,20 @@ def test_all_zones_must_pass(self): def test_distance_is_measured_in_three_dimensions(self): # (3, 4, 0) is 5 Mpc away, not 3 or 4. - cat = catalogue([ - (1, MW, 200.0, 10.0, 10.0, 10.0), - (2, 1e14, 900.0, 13.0, 14.0, 10.0), - ]) - assert len(candidates.select( - cat, RANGE, [ExclusionZone(distance=4.9, mass=7e13)], hubble=1.0)) == 1 - assert len(candidates.select( - cat, RANGE, [ExclusionZone(distance=5.1, mass=7e13)], hubble=1.0)) == 0 + cat = catalogue( + [ + (1, MW, 200.0, 10.0, 10.0, 10.0), + (2, 1e14, 900.0, 13.0, 14.0, 10.0), + ] + ) + assert ( + len(candidates.select(cat, RANGE, [ExclusionZone(distance=4.9, mass=7e13)], hubble=1.0)) + == 1 + ) + assert ( + len(candidates.select(cat, RANGE, [ExclusionZone(distance=5.1, mass=7e13)], hubble=1.0)) + == 0 + ) class TestRelativeExclusionZones: @@ -132,29 +152,35 @@ def test_excludes_the_candidate_itself(self): assert list(candidates.select(cat, RANGE, zones, hubble=1.0)["id"]) == [1] def test_rejects_a_comparable_mass_neighbour(self): - cat = catalogue([ - (1, MW, 200.0, 10.0, 10.0, 10.0), - (2, 0.8 * MW, 190.0, 11.0, 10.0, 10.0), # 1 Mpc, > half the mass - ]) + cat = catalogue( + [ + (1, MW, 200.0, 10.0, 10.0, 10.0), + (2, 0.8 * MW, 190.0, 11.0, 10.0, 10.0), # 1 Mpc, > half the mass + ] + ) zones = [ExclusionZone(distance=1.4, mass_multiple=0.5)] assert len(candidates.select(cat, RANGE, zones, hubble=1.0)) == 0 def test_ignores_a_much_lighter_neighbour(self): - cat = catalogue([ - (1, MW, 200.0, 10.0, 10.0, 10.0), - (2, 0.1 * MW, 90.0, 11.0, 10.0, 10.0), # 1 Mpc, well under half - ]) + cat = catalogue( + [ + (1, MW, 200.0, 10.0, 10.0, 10.0), + (2, 0.1 * MW, 90.0, 11.0, 10.0, 10.0), # 1 Mpc, well under half + ] + ) zones = [ExclusionZone(distance=1.4, mass_multiple=0.5)] assert list(candidates.select(cat, RANGE, zones, hubble=1.0)["id"]) == [1] def test_threshold_scales_with_each_candidate(self): # The 2e12 halo is more than half of 3e12 but the 1e12 halo is not, # so only the heavier candidate is rejected by its neighbour. - cat = catalogue([ - (1, 1e12, 200.0, 10.0, 10.0, 10.0), - (2, 3e12, 300.0, 40.0, 40.0, 40.0), - (3, 2e12, 250.0, 41.0, 40.0, 40.0), - ]) + cat = catalogue( + [ + (1, 1e12, 200.0, 10.0, 10.0, 10.0), + (2, 3e12, 300.0, 40.0, 40.0, 40.0), + (3, 2e12, 250.0, 41.0, 40.0, 40.0), + ] + ) zones = [ExclusionZone(distance=1.4, mass_multiple=0.5)] out = candidates.select(cat, RANGE, zones, hubble=1.0) assert list(out["id"]) == [1] @@ -170,18 +196,22 @@ class TestDefaultCaterpillarCriteria: ] def test_an_isolated_milky_way_analogue_is_selected(self): - cat = catalogue([ - (100, 1.5e12, 250.0, 50.0, 50.0, 50.0), - (200, 1e14, 900.0, 70.0, 50.0, 50.0), # 20 Mpc: clears all zones - ]) + cat = catalogue( + [ + (100, 1.5e12, 250.0, 50.0, 50.0, 50.0), + (200, 1e14, 900.0, 70.0, 50.0, 50.0), # 20 Mpc: clears all zones + ] + ) out = candidates.select(cat, RANGE, self.ZONES, hubble=1.0) assert list(out["id"]) == [100] def test_a_cluster_neighbour_at_3_mpc_disqualifies(self): - cat = catalogue([ - (100, 1.5e12, 250.0, 50.0, 50.0, 50.0), - (200, 1e14, 900.0, 53.0, 50.0, 50.0), # inside the 4 Mpc zone - ]) + cat = catalogue( + [ + (100, 1.5e12, 250.0, 50.0, 50.0, 50.0), + (200, 1e14, 900.0, 53.0, 50.0, 50.0), # inside the 4 Mpc zone + ] + ) assert len(candidates.select(cat, RANGE, self.ZONES, hubble=1.0)) == 0 def test_returns_columns_in_the_documented_order(self): @@ -201,22 +231,24 @@ def test_rvir_is_not_hubble_scaled(self): class TestCounts: def test_reports_base_sample_and_per_zone_populations(self): - cat = catalogue([ - (1, 1e12, 200.0, 10.0, 10.0, 10.0), - (2, 2e12, 250.0, 20.0, 20.0, 20.0), - (3, 1e13, 500.0, 30.0, 30.0, 30.0), - (4, 1e14, 900.0, 40.0, 40.0, 40.0), - ]) + cat = catalogue( + [ + (1, 1e12, 200.0, 10.0, 10.0, 10.0), + (2, 2e12, 250.0, 20.0, 20.0, 20.0), + (3, 1e13, 500.0, 30.0, 30.0, 30.0), + (4, 1e14, 900.0, 40.0, 40.0, 40.0), + ] + ) zones = [ ExclusionZone(distance=1.4, mass_multiple=0.5), ExclusionZone(distance=3.0, mass=7e12), ExclusionZone(distance=4.0, mass=7e13), ] result = candidates.counts(cat, RANGE, zones, hubble=1.0) - assert result["base_sample"] == 2 # the 1e12 and 2e12 haloes - assert result["above_zone2"] == 2 # 1e13 and 1e14 - assert result["above_zone3"] == 1 # 1e14 only - assert "above_zone1" not in result # relative zones have no fixed count + assert result["base_sample"] == 2 # the 1e12 and 2e12 haloes + assert result["above_zone2"] == 2 # 1e13 and 1e14 + assert result["above_zone3"] == 1 # 1e14 only + assert "above_zone1" not in result # relative zones have no fixed count class TestSerialisation: diff --git a/tests/test_writers_gadget.py b/tests/test_writers_gadget.py index 0e92f4b..583ed1c 100644 --- a/tests/test_writers_gadget.py +++ b/tests/test_writers_gadget.py @@ -36,10 +36,7 @@ def test_gas_is_zero_for_dark_matter_only_runs(self): assert gadget.softenings(100.0, 11)["Gas"] == 0.0 def test_finer_levelmax_gives_smaller_softening(self): - assert ( - gadget.softenings(100.0, 12)["Halo"] - < gadget.softenings(100.0, 11)["Halo"] - ) + assert gadget.softenings(100.0, 12)["Halo"] < gadget.softenings(100.0, 11)["Halo"] def test_reproduces_the_hardcoded_defaults(self): # The Gadget tab shipped these as literal defaults for a 100 Mpc/h box @@ -103,32 +100,71 @@ def test_subfind_parameters_only_present_when_requested(self): def test_contains_every_parameter_the_gui_exposed(self): params = parse(gadget.param_file(**PARAM, include_subfind=True)) required = { - "InitCondFile", "OutputDir", "EnergyFile", "InfoFile", "TimingsFile", - "CpuFile", "RestartFile", "SnapshotFileBase", "OutputListFilename", - "TimebinFile", "ICFormat", "SnapFormat", "TimeLimitCPU", - "CpuTimeBetRestartFile", "ResubmitOn", "ResubmitCommand", "MaxMemSize", - "PartAllocFactor", "BufferSize", "TimeBegin", "TimeMax", - "ComovingIntegrationOn", "PeriodicBoundariesOn", "CoolingOn", - "StarformationOn", "Omega0", "OmegaLambda", "OmegaBaryon", - "HubbleParam", "BoxSize", "OutputListOn", "TimeBetSnapshot", - "TimeOfFirstSnapshot", "TimeBetStatistics", "NumFilesPerSnapshot", - "NumFilesWrittenInParallel", "TypeOfTimestepCriterion", - "ErrTolIntAccuracy", "CourantFac", "MaxRMSDisplacementFac", - "MaxSizeTimestep", "MinSizeTimestep", "InitGasTemp", "MinGasTemp", - "TypeOfOpeningCriterion", "ErrTolTheta", "ErrTolForceAcc", - "TreeDomainUpdateFrequency", "DesNumNgb", "MaxNumNgbDeviation", - "UnitLength_in_cm", "UnitMass_in_g", "UnitVelocity_in_cm_per_s", - "GravityConstantInternal", "MinGasHsmlFractional", "ArtBulkViscConst", - "ErrTolThetaSubfind", "DesLinkNgb", + "InitCondFile", + "OutputDir", + "EnergyFile", + "InfoFile", + "TimingsFile", + "CpuFile", + "RestartFile", + "SnapshotFileBase", + "OutputListFilename", + "TimebinFile", + "ICFormat", + "SnapFormat", + "TimeLimitCPU", + "CpuTimeBetRestartFile", + "ResubmitOn", + "ResubmitCommand", + "MaxMemSize", + "PartAllocFactor", + "BufferSize", + "TimeBegin", + "TimeMax", + "ComovingIntegrationOn", + "PeriodicBoundariesOn", + "CoolingOn", + "StarformationOn", + "Omega0", + "OmegaLambda", + "OmegaBaryon", + "HubbleParam", + "BoxSize", + "OutputListOn", + "TimeBetSnapshot", + "TimeOfFirstSnapshot", + "TimeBetStatistics", + "NumFilesPerSnapshot", + "NumFilesWrittenInParallel", + "TypeOfTimestepCriterion", + "ErrTolIntAccuracy", + "CourantFac", + "MaxRMSDisplacementFac", + "MaxSizeTimestep", + "MinSizeTimestep", + "InitGasTemp", + "MinGasTemp", + "TypeOfOpeningCriterion", + "ErrTolTheta", + "ErrTolForceAcc", + "TreeDomainUpdateFrequency", + "DesNumNgb", + "MaxNumNgbDeviation", + "UnitLength_in_cm", + "UnitMass_in_g", + "UnitVelocity_in_cm_per_s", + "GravityConstantInternal", + "MinGasHsmlFractional", + "ArtBulkViscConst", + "ErrTolThetaSubfind", + "DesLinkNgb", } for name in ("Gas", "Halo", "Disk", "Bulge", "Stars", "Bndry"): required |= {f"Softening{name}", f"Softening{name}MaxPhys"} assert required - set(params) == set() def test_explicit_softening_overrides_the_derived_one(self): - custom = dict.fromkeys( - ("Gas", "Halo", "Disk", "Bulge", "Stars", "Bndry"), 1.5 - ) + custom = dict.fromkeys(("Gas", "Halo", "Disk", "Bulge", "Stars", "Bndry"), 1.5) params = parse(gadget.param_file(**PARAM, softening=custom)) assert float(params["SofteningHalo"]) == 1.5 @@ -172,9 +208,16 @@ def test_pmgrid_is_emitted_enabled(self): def test_options_always_on(self): on = self.enabled(gadget.config_sh()) - for token in ("PERIODIC", "UNEQUALSOFTENINGS", "PEANOHILBERT", - "WALLCLOCK", "MYSORT", "DOUBLEPRECISION", - "DOUBLEPRECISION_FFTW", "PLACEHIGHRESREGION=2"): + for token in ( + "PERIODIC", + "UNEQUALSOFTENINGS", + "PEANOHILBERT", + "WALLCLOCK", + "MYSORT", + "DOUBLEPRECISION", + "DOUBLEPRECISION_FFTW", + "PLACEHIGHRESREGION=2", + ): assert token in on def test_options_always_off(self): @@ -194,8 +237,7 @@ def test_subfind_enables_fof_and_subfind_together(self): assert "FOF_PRIMARY_LINK_TYPES=2" in on def test_secondary_link_types_are_configurable(self): - text = gadget.config_sh(include_subfind=True, - fof_secondary_link_types="4+8") + text = gadget.config_sh(include_subfind=True, fof_secondary_link_types="4+8") assert "FOF_SECONDARY_LINK_TYPES=4+8" in self.enabled(text) def test_disabled_options_stay_discoverable_as_comments(self): diff --git a/tests/test_writers_music.py b/tests/test_writers_music.py index b64cae0..f5f127d 100644 --- a/tests/test_writers_music.py +++ b/tests/test_writers_music.py @@ -64,16 +64,16 @@ def test_reproduces_the_original_derivation(self): # Pinned against the original inline arithmetic: delta starts at nrvir # and is updated as delta += delta*2 + 1 before each level is emitted. assert music.refinement_seeds(9, 11, 190897, 4, 34567) == [ - (9, 34567), (10, 190937), (11, 191018), + (9, 34567), + (10, 190937), + (11, 191018), ] class TestResimConf: def test_is_valid_ini_with_the_expected_sections(self): parser = parse(music.resim_conf(**RESIM)) - assert set(parser.sections()) == { - "setup", "cosmology", "random", "output", "poisson" - } + assert set(parser.sections()) == {"setup", "cosmology", "random", "output", "poisson"} def test_setup_values(self): setup = parse(music.resim_conf(**RESIM))["setup"] @@ -99,9 +99,9 @@ def test_ellipsoid_region_omits_ref_center_and_extent(self): assert "ref_extent" not in setup def test_box_region_includes_ref_center_and_extent(self): - kwargs = dict(RESIM, region="box", - ref_center=(0.51, 0.52, 0.53), - ref_extent=(0.21, 0.22, 0.23)) + kwargs = dict( + RESIM, region="box", ref_center=(0.51, 0.52, 0.53), ref_extent=(0.21, 0.22, 0.23) + ) setup = parse(music.resim_conf(**kwargs))["setup"] assert setup["ref_center"] == "0.51,0.52,0.53" assert setup["ref_extent"] == "0.21,0.22,0.23" @@ -153,9 +153,7 @@ def test_always_writes_ref_center_and_extent(self): assert setup["ref_center"] == "0.5, 0.5, 0.5" def test_has_no_region_point_file(self): - assert "region_point_file" not in parse( - music.parent_conf(**self.PARENT) - )["setup"] + assert "region_point_file" not in parse(music.parent_conf(**self.PARENT))["setup"] GOLDEN_RESIM = """\ diff --git a/tests/test_writers_submit.py b/tests/test_writers_submit.py index 8dd8324..f2f345b 100644 --- a/tests/test_writers_submit.py +++ b/tests/test_writers_submit.py @@ -11,14 +11,11 @@ def test_plain_run(self): def test_subfind_run_uses_the_sub_binary_param_and_logs(self): assert submit.mpirun_command(ranks=64, include_subfind=True) == ( - "mpirun -np 64 ./P-Gadget3_sub ./param_sub.txt 3 63 " - "1>OUTPUTsub 2>ERRORsub" + "mpirun -np 64 ./P-Gadget3_sub ./param_sub.txt 3 63 1>OUTPUTsub 2>ERRORsub" ) def test_subfind_snapshot_is_configurable(self): - cmd = submit.mpirun_command( - ranks=8, include_subfind=True, subfind_snapshot=127 - ) + cmd = submit.mpirun_command(ranks=8, include_subfind=True, subfind_snapshot=127) assert " 3 127 " in cmd def test_ranks_accept_strings_without_concatenating(self): @@ -62,12 +59,8 @@ def test_starts_with_bash_shebang(self): assert submit.slurm_script(**self.KWARGS).startswith("#!/bin/bash\n") def test_changes_into_the_run_directory_before_executing(self): - lines = [ - ln for ln in submit.slurm_script(**self.KWARGS).splitlines() if ln.strip() - ] - assert lines.index(f"cd {self.KWARGS['work_dir']}") < lines.index( - self.KWARGS["execute"] - ) + lines = [ln for ln in submit.slurm_script(**self.KWARGS).splitlines() if ln.strip()] + assert lines.index(f"cd {self.KWARGS['work_dir']}") < lines.index(self.KWARGS["execute"]) def test_mail_directives_omitted_without_an_email(self): text = submit.slurm_script(**dict(self.KWARGS, email=None)) @@ -103,8 +96,11 @@ def test_log_paths_are_absolute(self): def test_extra_flags_and_lines_are_included(self): text = submit.pbs_script( - **dict(self.KWARGS, extra_flags="#PBS -m ae", - extra_lines="source /opt/torque/etc/openmpi-setup.sh") + **dict( + self.KWARGS, + extra_flags="#PBS -m ae", + extra_lines="source /opt/torque/etc/openmpi-setup.sh", + ) ) assert "#PBS -m ae" in text assert "source /opt/torque/etc/openmpi-setup.sh" in text @@ -117,13 +113,9 @@ def test_extra_fields_omitted_when_blank(self): class TestDispatch: def test_selects_by_name_case_insensitively(self): common = dict(job_name="j", work_dir="/w", execute="run") - slurm = submit.script( - "slurm", **common, cores=8, minutes=10, memory_mb=100, queue="q" - ) + slurm = submit.script("slurm", **common, cores=8, minutes=10, memory_mb=100, queue="q") assert slurm.startswith("#!/bin/bash") - pbs = submit.script( - "PBS", **common, cores_per_node=8, nodes=1, queue="q" - ) + pbs = submit.script("PBS", **common, cores_per_node=8, nodes=1, queue="q") assert pbs.startswith("#!/bin/csh") def test_rejects_unknown_scheduler(self): From d68cbfc70c4ba90054b83767ac8aa6110c89cfd3 Mon Sep 17 00:00:00 2001 From: Brendan Griffen Date: Mon, 27 Jul 2026 22:58:34 +1000 Subject: [PATCH 17/19] docs: update README and AUDIT for the reorganised codebase The README still described a Python 2 package with no manifest, no tests and no license, run as `python main.py` from a flat directory. Bring it in line: - Python 3.9+, MIT, 171 tests in the badges - the status callout now distinguishes what works from what does not: the display-free library is pip-installable and tested, while the GUI still cannot start from a clean checkout because the `modules` I/O package is not distributed here - installation via the base / gui / dev extras, and `cme` as the entry point - a Library section showing the extracted API, since that is the part a reader can actually use - a Cluster profiles section documenting clusters.toml and its lookup order - the project structure and testing sections rewritten for the new layout - Known limitations replaced: the old entries were the defects this branch fixed. The honest remaining ones are the absent `modules` package, the complete lack of GUI test coverage, wxPython being legacy, and two parent paths still hardcoded in the ICs tab. AUDIT.md gains a Resolution status section: a table mapping each original finding to how it was addressed, a list of the twelve further defects found while doing the work, and eight outstanding items with the reasons they were deferred. The findings themselves are kept in full -- they are the rationale for the changes and the reference the regression tests cite. Also corrects one count in the duplication inventory: the mpirun execute-line builder had 18 copies across 6 handlers, not 12. The original figure counted lines rather than occurrences. --- AUDIT.md | 100 +++++++++++++++++++++++++++- README.md | 190 ++++++++++++++++++++++++++++++++++++++---------------- 2 files changed, 233 insertions(+), 57 deletions(-) diff --git a/AUDIT.md b/AUDIT.md index 649d3f7..1cf7167 100644 --- a/AUDIT.md +++ b/AUDIT.md @@ -1,7 +1,15 @@ # CME codebase audit Assessment of the repository as of 2026-07-27, covering 11 Python modules (~180 KB, -~4,000 lines) plus 13 screenshots. Line references are to the current `master` (`6067f9f`). +~4,000 lines) plus 13 screenshots. Line references are to the pre-uplift `master` +(`6067f9f`); the modules have since moved into `src/cme/` and been renamed, so use the +descriptions rather than the line numbers to locate anything. + +> [!NOTE] +> **Most of this has now been addressed** on the `uplift/modernise-and-fix` branch. See +> [Resolution status](#resolution-status) at the end for what was fixed, what was found +> during the work, and what is still outstanding. The findings are kept in full because +> they are the rationale for the changes and the reference for the regression tests. ## Verdict @@ -115,7 +123,7 @@ layout proposed in section 6. | `dircheck = ... + "/outputs/groups_0" + snapnum` followed by an existence test | 9 | `contam.py:209-284,426` | | The `if self.projopt == 'xy' / 'xz' / 'yz'` axis-selection block | 3 | `ics.py:391-419,462-490,613-641` | | The 12-line softening cascade | 2 | `gadgetrun.py:522-533,543-554` | -| The `mpirun -np ... ./P-Gadget3 ...` execute-line builder | 12 | `gadgetrun.py:415-453,928-929` | +| The `mpirun -np ... ./P-Gadget3 ...` execute-line builder | 18 copies across 6 handlers, plus 2 in `__init__` | `gadgetrun.py:415-453,928-929` | | The hardcoded Hubble parameter `0.6711` | 6 | `candidates.py:151,305`, `contam.py:93,316,385`, `halos.py:400` | | Near-identical `_*_changed` handlers that clear the figure | 4 | `mergertree.py:85-123` | @@ -327,3 +335,91 @@ Stop there. P1 onward is only worth it if the tool is to be used again. | Tests | 0 | | Docstrings | 1 | | Largest modules | `gadgetrun.py` (49 KB), `ics.py` (47 KB) | + + +--- + +## Resolution status + +Addressed on `uplift/modernise-and-fix`, in 17 commits. 171 tests now cover the +display-free library. + +### Fixed + +| Finding | How | +| --- | --- | +| 1.2 Python 2 only | Ported: 35 print statements, 6 `xrange`, 21 pandas `.ix` → `.loc`, five invalid escapes. All modules parse on 3.9-3.12. | +| 1.3 Retired `enthought.*` namespace | Mapped onto `traits` / `traitsui` / `enable`. wxPython kept as the toolkit. | +| 2.1.1 SLURM `TypeError` | `cme.writers.submit` formats rank counts with `int()`. Regression test. | +| 2.1.2 Gadget sweep inert (`nvir` typo) | Corrected to `nrvir`. | +| 2.1.3 Four `makeactive ==` no-ops | Removed with the handler dedup in the contamination tab. | +| 2.1.4 `gethostid` `UnboundLocalError` | Returns `None`; tolerates a missing summary file. | +| 2.1.5 `gethalos_xy` `NameError` / unbound return | Restructured; returns empty arrays on a miss. | +| 2.1.6 Plot button inert in parent mode | Plotting block de-indented out of the zoom branch. | +| 2.1.7 Odyssey preset never applied (`clustopt` typo) | Replaced wholesale by `clusters.toml`. | +| 2.1.8 `Clear` wrote to the wrong object | One `_publish_sample()` for all five consumers. | +| 2.1.9 Three MUSIC settings never applied | Corrected names; `presmooth`/`postsmooth` are now real traits. | +| 2.1.10 `nhalo` unbound | Vectorised lookup that reports the miss. | +| 2.1.11 `determineboolstr` unbound return | Replaced by a total function in `cme.writers.music`. | +| 2.1.12 Run script opened before its directory | `writers.write` creates parents, called once the path is known. | +| 2.2.1 3D distribution plotted y twice | `z = tmphalos['posZ']`. | +| 2.2.2 `hostposx` reported z | `host['posX']`. | +| 2.2.3 Phantom zero-mass candidate | `select()` returns an empty array; the add-halo guard no longer tests for the sentinel. | +| 2.2.4 `NRVIR['1']` in displayed paths | Routed through `naming.lagr_region_path`. | +| 2.2.5 SLURM ranks from PBS core count | One `execute_command()` per scheduler. | +| 2.2.6 SUBFIND ignored on SLURM | Same. | +| 2.3 Dead code | `Analysis`, duplicate `boxtype`, 3 unused imports, 5 unused buttons, 20 dead locals, ~40 lines of commented-out code removed. | +| 3.1-3.3 Shell injection (70 sites) | `cme.shell` runs argument lists; no `shell=True` or `os.system` remains. The `tail -n+96` splice is done in Python. | +| 3.4 19 personal identifiers | Moved into `clusters.toml`. | +| 3.5-3.6 Unprotected file handles | No raw `open()` outside `cme.writers`; existence checks use `os.path.exists`. | +| 4.1 Science trapped in event handlers | `cme.naming`, `cme.cosmology`, `cme.science`, `cme.writers` are display-free and tested. | +| 4.2 Duplication | 18 folder-name builders → `cme.naming`; 6 Hubble literals → `cme.cosmology`; 18 mpirun strings → one method; softening cascade, projection block and 4 identical handlers all collapsed. | +| 4.3 Star imports | Explicit imports plus `__all__` in `cme/ui/common.py`. The accidental `patches` dependency is now explicit. | +| 4.5 Performance | Parent snapshot reads hoisted out of the nrvir loop (~30 GB of I/O for a five-value sweep); candidate extraction made O(N); `vstack`-per-match replaced. | +| 5 Hygiene | `pyproject.toml`, `LICENSE`, `.gitignore`, CI, 171 tests, docstrings on the library. | + +### Found during the work, also fixed + +- Snapshot indices were zero-padded by concatenation (`"groups_0" + str(n)`), correct only + for two-digit numbers. Snapshot 5 looked for `groups_05` where Gadget writes `groups_005`, + and the zoom tabs' default of 255 produced `snapdir_0255`. +- The contamination and halo tabs hardcoded `_Z127` rather than reading their own `zinit`, + so a run generated at another starting redshift was looked up in a directory that had + never been created. +- Base paths were joined as `self.gadpath + 'halos/H'`, silently producing + `.../datahalos/H190897` whenever the path trait lacked a trailing slash. +- `for lmini in self.lmin` iterated an Enum holding a *string*, so a levelmin of 10 or above + ran the sweep twice, with levelmin `'1'` and then `'0'`. +- `HubbleParam` was stored as H0 in a trait named for h and divided by 100 on the way out, + so the value displayed in the GUI disagreed with the value written to disk. +- `Config.sh` emitted `TOKEN# comment` with no separating space for tokens longer than the + comment column. +- The Gadget existence table hardcoded snapshot 63 regardless of the requested output count. +- `ApplicationMain.__init__` never called `HasTraits.__init__`, so trait initialisation was + skipped and kwargs were discarded. +- `_clusteropt_changed` and `_username_changed` derived *different* path sets from the same + inputs, so which paths you got depended on the order you touched the fields. +- `reWriteIC` was imported as `re`, shadowing the standard library module. + +### Outstanding + +1. **The `modules` I/O package is still absent** (finding 1.1). Nothing else in this list + matters as much: the GUI cannot start without it. Vendor or pin it. +2. **The GUI has no test coverage.** The library is well covered; the tabs are verified only + by `compileall` and inspection, because exercising them needs a display and a real parent + simulation. A Qt port would make headless widget tests feasible. +3. **wxPython → Qt** (finding 1.4). Deliberately deferred: it is a large diff that cannot be + verified here, and it would have obscured the behavioural fixes. +4. **The Install tab** (finding 1.5) still expects a `lib/installs/` source tree that is not + in the repository. It is hardened but should probably be deleted in favour of the system + package manager. +5. **`self.main` coupling** (finding 4.4). Tabs still reach through a back-reference to + mutate siblings, so construction order remains load-bearing. +6. **`PARENT_SNAPSHOT` / `PARENT_ICS`** in `cme/ui/ics.py` are still hardcoded to the + Caterpillar 512³ parent; they belong in `clusters.toml`. +7. **The non-PLANCK cosmologies** in `cme.cosmology` were taken from the cited papers, not + from the original `grifflib`. PLANCK cross-checks exactly against the values that were + hardcoded in the Gadget tab; the others should be diffed against `grifflib` if it is + recovered, before being used for new science. +8. **TraitsUI view declarations** are still deeply nested (up to eight redundant group + levels) and unformatted, and `cme/ui/` is excluded from the `ruff format` check. diff --git a/README.md b/README.md index 47c7080..fa4290d 100644 --- a/README.md +++ b/README.md @@ -4,11 +4,11 @@ An interactive desktop GUI for setting up, running and inspecting cosmological zoom-in simulations. -![Python](https://img.shields.io/badge/python-2.7-blue) -![Status](https://img.shields.io/badge/status-archived-lightgrey) +![Python](https://img.shields.io/badge/python-3.9%2B-blue) +![Tests](https://img.shields.io/badge/tests-171-brightgreen) ![GUI](https://img.shields.io/badge/GUI-Traits%20%2B%20wxPython-orange) ![Viz](https://img.shields.io/badge/3D-Mayavi-8a2be2) -![License](https://img.shields.io/badge/license-none%20specified-red) +![License](https://img.shields.io/badge/license-MIT-green) @@ -19,14 +19,17 @@ suite of resimulations can be configured and launched without writing code. It w the [Caterpillar Project](https://www.caterpillarproject.org) at MIT/Harvard, but the workflow applies to any suite built on MUSIC + Gadget + Rockstar. -> [!WARNING] -> **Status: archived / historical.** Last substantive development was 2014. This is -> **Python 2 only** and targets the retired pre-2011 `enthought.*` Traits namespace with a -> wxPython backend — it will not import on Python 3 or on any current ETS release. It also -> depends on an I/O package that is **not included in this repository** -> (see [Prerequisites](#prerequisites)). Treat it as a reference implementation and a record -> of the Caterpillar setup workflow. See [`AUDIT.md`](AUDIT.md) for a full assessment and a -> modernisation plan. +> [!IMPORTANT] +> **The GUI cannot run from a clean checkout.** Every tab imports a `modules` package of +> simulation I/O readers that is **not distributed with this repository** — see +> [Prerequisites](#prerequisites). Launching the interface also needs a display, a built +> MUSIC and P-Gadget3, and Rockstar catalogues for a parent simulation. +> +> What *does* work standalone is everything under [Library](#library): the path +> conventions, the cosmologies, the candidate selection, and the MUSIC/Gadget/scheduler +> file writers are display-free, `pip`-installable and covered by 171 tests. The code was +> Python 2 until recently; see [`AUDIT.md`](AUDIT.md) for the assessment that drove the +> modernisation and what remains outstanding. --- @@ -88,16 +91,17 @@ Cosmologies available throughout: WMAP1, WMAP3, WMAP5, WMAP7, WMAP9, PLANCK. ### Python environment -Python 2.7 with the Enthought Python Distribution (EPD) or a comparable ETS 3.x stack: +Python 3.9 or newer. Dependencies are declared in `pyproject.toml` and split by what +you need: -- `enthought.traits` / `enthought.traits.ui` — the ETS 3.x namespace, **not** the modern `traits` / `traitsui` packages -- `traitsui` — for `ArrayViewEditor` and `TabularEditor` -- `wxPython` — the TraitsUI and matplotlib backend used here -- `mayavi` + `tvtk` — 3D scenes -- `matplotlib` (with the `WXAgg` backend), `numpy`, `pandas` +| Extra | Contents | Needed for | +| --- | --- | --- | +| *(base)* | `numpy`, `pandas` | the display-free library and the test suite | +| `gui` | `traits`, `traitsui`, `enable`, `mayavi`, `matplotlib`, `wxPython` | launching the interface | +| `dev` | `pytest`, `ruff` | running tests and linting | -There is no dependency manifest in the repository; the list above is derived from -`Common.py` and the per-tab imports. +The GUI uses the modern `traits` / `traitsui` packages with the wxPython toolkit. A Qt +port is the obvious next step but has not been done — see [`AUDIT.md`](AUDIT.md). ### External I/O package (not included) @@ -113,7 +117,7 @@ runs. It provides: | `modules.readhalos.readsubf` | SUBFIND group catalogue reader (`subfind_catalog`) | | `modules.mergertrees.MTCatalogue` | `consistent-trees` merger tree reader | | `modules.convertfiles.reWriteIC` | `getBlocks()` — post-processes MUSIC output for Gadget | -| `modules.brendanlib.grifflib` | Helpers: `cosmoconstant`, `getcandidatelist`, `getcentext`, `getlagrxyz`, `drawcircle`, `addsubtitle`, `placenormtext`, `CorrectPos`, `COM` | +| `modules.brendanlib.grifflib` | Helpers: `getcandidatelist`, `getcentext`, `getlagrxyz`, `drawcircle`, `addsubtitle`, `placenormtext`, `CorrectPos`, `COM`. (`cosmoconstant` is no longer needed — see `cme.cosmology`.) | ### External codes and data @@ -131,27 +135,30 @@ runs. It provides: git clone https://github.com/bgriffen/cme.git cd cme -# The I/O package is a separate dependency — put its parent directory on PYTHONPATH +# The display-free library and the test suite: +pip install -e ".[dev]" +python -m pytest tests/ + +# To also launch the interface: +pip install -e ".[gui]" +# The simulation I/O readers are a separate, undistributed dependency: export PYTHONPATH=/path/to/modules-parent:$PYTHONPATH ``` -No `pip install` path exists: there is no `setup.py` or `pyproject.toml`, and the package -is run in place from the repository root. - ## Usage ```bash -python main.py +cme # or: python -m cme.app ``` The window splits into a plot panel (matplotlib, plus a Mayavi scene on its own tab) on the left and the workflow tabs on the right. Marker colour, style and size at the bottom left apply to every 2D plot. Work left to right through the tabs: -1. **Home** — set your username and the paths to your home directory, MUSIC, P-Gadget3, the - parent simulation and the project data directory. Everything downstream derives from - these. Cluster presets are keyed off `platform.node()`, so an unrecognised machine needs - its paths entered by hand. +1. **Home** — pick a cluster profile and confirm the derived paths to MUSIC, P-Gadget3, + the parent simulation and the project data directory. Everything downstream derives from + these. Profiles come from [`clusters.toml`](#cluster-profiles) and are matched against + the hostname automatically. 2. **Candidates** — either load an existing `candidates.dat` or run the selection, then add the IDs you want into the working halo sample. That sample propagates to every other tab. 3. **Initial Conditions** — generate and inspect the Lagrangian region per halo and `nrvir` @@ -178,41 +185,115 @@ alongside a `.head` file holding the region centre and extent. ``` cme/ -├── main.py # Application entry point; assembles the tabbed window -├── Common.py # Shared imports and the matplotlib-in-Traits figure editor -├── header.py # Home tab — cluster detection and base paths -├── install.py # Install tab — builds FFTW/HDF5/GSL/zlib/szip -├── candidates.py # Candidates tab — isolation-criteria halo selection -├── ics.py # Initial Conditions tab — Lagrangian regions and MUSIC configs -├── gadgetrun.py # Gadget tab — param.txt, Config.sh, PBS/SLURM submission -├── halos.py # Halo Finder tab — Rockstar catalogue plots and 3D scenes -├── mergertree.py # Merger Tree tab — main-branch evolution plots -├── contam.py # Contam. tab — low-resolution particle contamination -└── screenshots/ # Interface screenshots used in this README +├── clusters.toml # Cluster profiles: hostnames, paths, scheduler settings +├── pyproject.toml +├── src/cme/ +│ ├── app.py # Entry point; assembles the tabbed window +│ ├── naming.py # Run-directory and Lagrangian path conventions +│ ├── cosmology.py # Named cosmologies (WMAP1-9, PLANCK) +│ ├── config.py # Loads clusters.toml +│ ├── shell.py # Runs external commands without a shell +│ ├── science/ +│ │ └── candidates.py # Isolation-criteria halo selection +│ ├── writers/ +│ │ ├── music.py # MUSIC .conf generation +│ │ ├── gadget.py # param.txt, Config.sh, ExpansionList, softenings +│ │ └── submit.py # PBS and SLURM submission scripts +│ └── ui/ # One HasTraits tab per module +│ ├── common.py # Shared imports, matplotlib-in-TraitsUI editor +│ ├── home.py install.py candidates.py ics.py +│ └── gadget.py halos.py mergertree.py contamination.py +├── tests/ +└── screenshots/ ``` Each tab is a `HasTraits` subclass instantiated by `ApplicationMain` and handed a reference to it, so tabs share state (the halo sample, the base paths, the active figure) through the `self.main` back-reference. +## Library + +Everything outside `cme/ui/` is display-free: plain functions over arrays and scalars, with +no Traits, matplotlib or filesystem dependency. That makes the pipeline scriptable for a +large suite and testable without a cluster. + +```python +from cme import naming, cosmology +from cme.writers import music, gadget, submit +from cme.science.candidates import select, ExclusionZone + +# The run-directory convention, in one place +naming.run_dir_name(halo_id=190897, region="ellipsoid", padding=7, + levelmin=7, levelmax=11, overlap=4, nrvir=4) +# 'H190897_BE_Z127_P7_LN7_LX11_O4_NV4' + +# Isolation criteria: "no halo above mass M within distance R" +select(hosts, mass_range=(7e11, 7e12), hubble=0.6711, zones=[ + ExclusionZone(distance=1.4, mass_multiple=0.5), # half my mass, 1.4 Mpc + ExclusionZone(distance=3.0, mass=7e12), + ExclusionZone(distance=4.0, mass=7e13), +]) + +# Configuration files as strings, so they can be diffed and asserted on +music.resim_conf(cosmo=cosmology.PLANCK, boxlength=100, levelmax=11, ...) +gadget.param_file(cosmo=cosmology.PLANCK, boxsize=100.0, levelmax=11) +submit.slurm_script(job_name="H19EP7L11N4", cores=512, ...) +``` + +## Cluster profiles + +`clusters.toml` describes each machine: which hostnames select it, where home directories +live, and where MUSIC, P-Gadget3 and the project data sit, using `{home}`, `{user}` and +`{master}` placeholders. + +```toml +[clusters.odyssey] +hostnames = ["rclogin13.rc.fas.harvard.edu"] +home_root = "/n/home01" +gadget = "{master}/lib/P-Gadget3" +data = "{home}/caterpillar" +scheduler = "SLURM" +queue = "itc_cluster" +``` + +Resolution order: `$CME_CLUSTERS`, then `~/.config/cme/clusters.toml`, then the copy in the +repository. A profile whose `hostnames` list is empty must be selected by hand on the Home +tab. With no file found at all, a single `local` profile under your home directory is used, +so the application still starts. + ## Testing -There is no test suite, and no CI configuration. Adding one is the first recommendation in -[`AUDIT.md`](AUDIT.md); the pure-computation parts (candidate selection, softening scaling, -config-file writers, folder-name construction) are the natural place to start, since they -can be tested without a display or a cluster. +```bash +pip install -e ".[dev]" +python -m pytest tests/ # 171 tests +python -m ruff check . +``` + +The suite covers the display-free library only — the naming conventions, the cosmologies, +the candidate selection, the config-file writers and the cluster-profile loader — because +those are the parts that can run without a display, a cluster or the undistributed +`modules` package. Several tests are pinned regressions for specific defects; they cite the +relevant `AUDIT.md` section in a comment. + +CI additionally runs the doctests, `ruff format --check` over the library, and +`compileall` over the GUI modules on Python 3.9 and 3.12 — those cannot be imported in CI, +but they must at least parse. ## Known limitations -- Python 2 syntax and the retired `enthought.*` namespace: the code will not import on - Python 3 or on any current ETS release. -- Cluster and user paths are hardcoded in several places (`header.py`, `gadgetrun.py`), - including specific hostnames, usernames and email addresses. Anyone else must edit source. -- Shell commands are assembled by string concatenation and run with `shell=True`, so paths - containing spaces or shell metacharacters will break or misbehave. -- The SLURM script writer concatenates integer traits with strings and raises `TypeError`; - only the PBS path is known-good. -- No dependency manifest, no packaging, no tests. +- **The GUI cannot be launched from a clean checkout**, because the `modules` I/O package is + not distributed here. This is the single biggest barrier to reuse. +- **The GUI is untested.** 171 tests cover the library; the tabs themselves have no + coverage, since exercising them needs a display and a real parent simulation. Changes to + `cme/ui/` are verified only by inspection and by `compileall`. +- wxPython is legacy. TraitsUI still supports it, but Qt (PySide6) is the practical default + and a port is the obvious next step. +- The Install tab builds FFTW2, HDF5 1.8 and GSL 1.9 from a `lib/installs/` source tree that + is also not in the repository. Use your system package manager instead. +- Two parent-simulation paths remain hardcoded in `cme/ui/ics.py` + (`PARENT_SNAPSHOT`, `PARENT_ICS`); they belong in `clusters.toml`. +- The tabs still share state through a `self.main` back-reference, which makes their + construction order load-bearing. ## Acknowledgments @@ -230,5 +311,4 @@ Brendan Griffen — [brendan.f.griffen@gmail.com](mailto:brendan.f.griffen@gmail ## License -No license file is present, so all rights are reserved by default. If this is meant to be -reusable, add a `LICENSE` — MIT or BSD-3-Clause is conventional for astronomy tooling. +MIT — see [`LICENSE`](LICENSE). From cfd24ceb6eddbf16773cc9646eea788852bc0c74 Mon Sep 17 00:00:00 2001 From: Brendan Griffen Date: Mon, 27 Jul 2026 23:00:42 +1000 Subject: [PATCH 18/19] fix: source the Gadget tab's cosmology defaults from cme.cosmology Two leftovers from the cosmology extraction, both concerning the meaning of HubbleParam. The tab's __init__ still carried Omega0, OmegaLambda and HubbleParam as literals, and _subscript_button_fired unpacked cosmoconstant() positionally -- which puts H0 (67.11) into HubbleParam. Now that the parameter file writer takes a Cosmology and emits the dimensionless h, that assignment left the displayed value disagreeing with the written one by a factor of 100, in the opposite direction to the original bug. Both now read the fields off the Cosmology object, so the table in cme.cosmology is the only place these numbers appear. --- src/cme/ui/gadget.py | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-) diff --git a/src/cme/ui/gadget.py b/src/cme/ui/gadget.py index 9df2c7e..719857d 100644 --- a/src/cme/ui/gadget.py +++ b/src/cme/ui/gadget.py @@ -295,7 +295,14 @@ def _subscript_button_fired(self): else: print("PATH FOUND:",filepath) self.levelmaxuse = int(self.lmax) - self.Omega0,self.OmegaLambda,self.OmegaBaryon,self.HubbleParam,sigma8,nspec = cosmology.cosmoconstant(self.cosmologylist[0]) + # Mirror the selected cosmology onto the displayed + # traits. This used to unpack cosmoconstant(), which + # put H0 into HubbleParam -- a field the parameter + # file writes as the dimensionless h. + cosmo = cosmology.get(self.cosmologylist[0]) + self.Omega0 = cosmo.omega_m + self.OmegaLambda = cosmo.omega_l + self.HubbleParam = cosmo.h # Written directly into the run directory. The # original wrote into the current working directory # and then shelled out to `mv`. @@ -674,10 +681,12 @@ def __init__(self, main, **kwargs): self.PeriodicBoundariesOn = 1 self.CoolingOn = 0 self.StarformationOn = 0 - self.Omega0 = 0.3175 # 0.276 - self.OmegaLambda = 0.6825 # 0.724 - self.OmegaBaryon = 0.0 - self.HubbleParam = 0.6711 # 0.703 + # Displayed defaults, from the same table the writer uses. HubbleParam is + # the dimensionless h, matching what the parameter file receives. + self.Omega0 = cosmology.DEFAULT.omega_m + self.OmegaLambda = cosmology.DEFAULT.omega_l + self.OmegaBaryon = 0.0 # dark-matter-only runs + self.HubbleParam = cosmology.DEFAULT.h self.BoxSize = 100.0 self.OutputListOn = 1 self.TimeBetSnapshot = 0.0 From adb3e1f40843f5da5bf0014c697dfc4fc896e680 Mon Sep 17 00:00:00 2001 From: Brendan Griffen Date: Mon, 27 Jul 2026 23:12:41 +1000 Subject: [PATCH 19/19] chore: replace personal contact details with a role address Points every contact reference at contact+cme@grifflabs.dev instead of a personal mailbox: the packaging metadata, the `__author__` string, and the README's contact section. Also removes the last two pieces of personal data in the tree: - clusters.toml pinned a site-specific login name on the barrine profile. That file is shared, so a login belongs in a personal override; the header now documents ~/.config/cme/clusters.toml for exactly this. {user} falls back to the running user's login, which is the sensible default anyway. - a commented-out line in the Gadget tab still carried an absolute path under one user's home directory. AUDIT.md's description of finding 3.4 no longer enumerates the actual logins, hostnames and home directories -- it makes the same point by counting them. Test changes that follow: - the profile-level `username` feature is still covered, but against an inline config rather than a shipped profile that should not pin one. - a new test asserts the shipped profiles resolve {user} to whoever is running. - the "no personal home directory" test previously grepped resolved paths for a specific surname. It now resolves each profile as two different users and asserts the results differ only where the username appears, which is the actual property wanted and does not need the name to state it. Left in place: the github.com/bgriffen/cme URLs, which are the repository's real location, and the LICENSE copyright holder, which needs to name a person. --- AUDIT.md | 2 +- README.md | 2 +- clusters.toml | 14 +++++++++++-- pyproject.toml | 2 +- src/cme/__init__.py | 2 +- src/cme/ui/gadget.py | 1 - tests/test_config.py | 49 ++++++++++++++++++++++++++++++++++---------- 7 files changed, 54 insertions(+), 18 deletions(-) diff --git a/AUDIT.md b/AUDIT.md index 1cf7167..997f55c 100644 --- a/AUDIT.md +++ b/AUDIT.md @@ -92,7 +92,7 @@ Grouped by confidence. | 3.1 | **Command injection / breakage via string-concatenated shell commands.** 70 call sites use `os.system` or `subprocess.call(..., shell=True)` with GUI-supplied paths interpolated in. A path containing a space silently splits; one containing `;` or `$(...)` executes. | `install.py` (all buttons), `ics.py:257,325`, `gadgetrun.py:302,306,309,362,368,370,406` | | 3.2 | **`rm` commands built from unvalidated paths.** `f1.write("rm " + str(filepath) + "/*.e*\n")` and `rm wnoise* temp*` run in a directory derived from GUI state. If a path trait is empty, the glob widens. | `gadgetrun.py:374-375`, `ics.py:322` | | 3.3 | **A fragile offset into a third-party file.** `tail -n+96 .../Config.sh` assumes the P-Gadget3 config has exactly 95 lines of preamble. Any upstream change silently corrupts the generated config. | `gadgetrun.py:359,365` | -| 3.4 | **Personal identifiers hardcoded in source** — 19 occurrences across usernames (`bgriffen`, `uqbgriff`), hostnames (`csr-dyn-150.mit.edu`, `bigbang.mit.edu`, `rclogin13.rc.fas.harvard.edu`), absolute paths (`/bigbang/data/`, `/n/home01/bgriffen/data/`, `/Users/griffen/Desktop/cme/`) and email addresses. Also `halos.py:131`, an absolute path baked into a method body. | `header.py:55-99`, `gadgetrun.py:9-10,123,257,263`, `halos.py:131` | +| 3.4 | **Personal identifiers hardcoded in source** — 19 occurrences: two cluster login names, four specific hostnames, three absolute paths under one user's home directory, and two email addresses. Also `halos.py:131`, an absolute path baked into a method body. | `header.py:55-99`, `gadgetrun.py:9-10,123,257,263`, `halos.py:131` | | 3.5 | **Every file handle is unprotected by `with`.** ~15 `open()`/`close()` pairs leak on exception, and `_getcandidates_button_fired` can leave `out` open on any error mid-loop. | `ics.py:666,676,743,837`, `gadgetrun.py:253,312,657,664`, `candidates.py:175` | | 3.6 | **`open()` used as an existence test**, in a `try`/`except IOError`, rather than `os.path.exists`. Race-prone and obscures intent. | `ics.py:374-378,450-451,515-519`, `gadgetrun.py:813-817` | diff --git a/README.md b/README.md index fa4290d..2d5843f 100644 --- a/README.md +++ b/README.md @@ -307,7 +307,7 @@ Developed as part of the Caterpillar Project at MIT/Harvard. ## Contact -Brendan Griffen — [brendan.f.griffen@gmail.com](mailto:brendan.f.griffen@gmail.com) +Brendan Griffen — [contact+cme@grifflabs.dev](mailto:contact+cme@grifflabs.dev) ## License diff --git a/clusters.toml b/clusters.toml index 6eb652c..5a0e572 100644 --- a/clusters.toml +++ b/clusters.toml @@ -13,7 +13,18 @@ # # `hostnames` lists the platform.node() values that select this profile # automatically. A profile with no hostnames must be selected by hand. -# Copy this file to ~/.config/cme/clusters.toml to override it. +# +# No profile here sets `username`, so {user} is your login name. If a site gives +# you a different login than the one you use locally, set it in a personal +# override rather than here -- this file is shared: +# +# ~/.config/cme/clusters.toml +# +# [clusters.barrine] +# username = "your-site-login" +# +# The lookup order is $CME_CLUSTERS, then ~/.config/cme/clusters.toml, then this +# file. default = "local" @@ -69,7 +80,6 @@ queue = "default" [clusters.barrine] hostnames = ["barrine"] -username = "uqbgriff" gadget = "{master}/lib/P-Gadget3" music = "{master}/lib/music" data = "{master}/projects/caterpillar/data" diff --git a/pyproject.toml b/pyproject.toml index a065f08..e82cb0e 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -9,7 +9,7 @@ description = "Caterpillar Made Easy: a GUI for setting up, running and inspecti readme = "README.md" requires-python = ">=3.9" license = { file = "LICENSE" } -authors = [{ name = "Brendan Griffen", email = "brendan.f.griffen@gmail.com" }] +authors = [{ name = "Brendan Griffen", email = "contact+cme@grifflabs.dev" }] keywords = ["cosmology", "simulation", "zoom-in", "gadget", "music", "rockstar"] classifiers = [ "Development Status :: 3 - Alpha", diff --git a/src/cme/__init__.py b/src/cme/__init__.py index f42c808..da77892 100644 --- a/src/cme/__init__.py +++ b/src/cme/__init__.py @@ -1,5 +1,5 @@ """ A tool for constructing and analysing cosmological simulations. """ -__author__ = "Brendan Griffen " +__author__ = "Brendan Griffen " __version__ = "0.1.0" diff --git a/src/cme/ui/gadget.py b/src/cme/ui/gadget.py index 719857d..ca0b633 100644 --- a/src/cme/ui/gadget.py +++ b/src/cme/ui/gadget.py @@ -351,7 +351,6 @@ def _subscript_button_fired(self): os.path.join(gadgetfilepath, "P-Gadget3", "Config.sh")) #try: - # with open("/bigbang/data/bgriffen/lib/P-Gadget3/P-Gadget3"): pass lines.append("cd " + str(filepath) + "\n") lines.append("rm " + str(filepath) + "/*.e*\n") lines.append("rm " + str(filepath) + "/*.o*\n") diff --git a/tests/test_config.py b/tests/test_config.py index e2ce252..8c0c5e2 100644 --- a/tests/test_config.py +++ b/tests/test_config.py @@ -88,14 +88,32 @@ def test_local_profile_uses_the_real_home_directory(self, cfg): paths = config.resolve("local", config=cfg, username="alice") assert paths["master"] == os.path.expanduser("~") - def test_profile_username_is_used_when_none_is_given(self, cfg): - # The barrine profile carries a site-specific login name, which used to - # be assigned from an if-chain in the Home tab. - assert config.resolve("barrine", config=cfg)["username"] == "uqbgriff" + def test_profile_username_is_used_when_none_is_given(self): + # A site may issue a login different from the local one. The shipped + # profiles deliberately do not set this -- it belongs in a personal + # override -- so the feature is exercised against an inline config. + cfg = { + "default": "site", + "clusters": { + "site": { + "hostnames": [], + "home_root": "/scratch", + "gadget": "{master}/lib/P-Gadget3", + "username": "site-login", + } + }, + } + resolved = config.resolve("site", config=cfg) + assert resolved["username"] == "site-login" + assert resolved["gadget"] == "/scratch/site-login/lib/P-Gadget3" + + # An explicit username still wins. + assert config.resolve("site", config=cfg, username="alice")["username"] == "alice" - def test_explicit_username_overrides_the_profile(self, cfg): - paths = config.resolve("barrine", config=cfg, username="alice") - assert paths["username"] == "alice" + def test_shipped_profiles_do_not_pin_a_login_name(self, cfg): + # clusters.toml is shared, so {user} must resolve to whoever is running. + for name in config.profile_names(cfg): + assert config.resolve(name, config=cfg, username="alice")["username"] == "alice" def test_scheduler_settings_are_exposed(self, cfg): assert config.resolve("odyssey", config=cfg)["scheduler"] == "SLURM" @@ -121,9 +139,18 @@ def test_profiles_contain_no_email_addresses(self): with open(REPO_CLUSTERS) as handle: assert "@" not in handle.read() - def test_no_profile_hardcodes_a_personal_home_directory(self, cfg): + def test_paths_differ_only_by_the_running_user(self, cfg): + """No third party's login or home directory can be baked into a path. + + Resolving the same profile as two different users must differ only where + the username appears. Anything constant across both is site structure; + anything varying is correctly attributed to the caller. + """ for name in config.profile_names(cfg): - paths = config.resolve(name, config=cfg, username="alice") + alice = config.resolve(name, config=cfg, username="alice") + bob = config.resolve(name, config=cfg, username="bob") for key in config.PATH_KEYS: - assert "bgriffen" not in paths[key], (name, key) - assert "griffen" not in paths[key], (name, key) + assert alice[key].replace("alice", "") == bob[key].replace("bob", ""), ( + name, + key, + )