From 82e73b10a913fa5b67dbc9f48eb6e227b2d17a7f Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Sun, 16 Aug 2026 18:15:50 +0000 Subject: [PATCH 1/4] fix: gate intensity state on CLI writers and docs Apply the intensity-state contract to estimate-k, subtract-buffer, and minimal_2d. Stamp 1/cm only for labeled absolute_cm^-1 data, refuse unlabeled I, and point reviewers at unreleased 2.0.0 on main. Co-authored-by: Delun Gong --- CHANGELOG.md | 18 ++ CONTRIBUTING.md | 6 +- README.md | 35 ++-- SASAbs.py | 58 +++++- SUBMISSION_READINESS.md | 20 +- docs/api.md | 45 +++-- docs/architecture.md | 12 +- docs/joss-submission-checklist.md | 12 +- docs/reviewer-faq.md | 30 ++- examples/k_measured.csv | 1 + examples/k_reference.csv | 2 + examples/manual-verification.md | 4 + examples/minimal_2d/README.md | 7 +- .../minimal_2d/run_minimal_2d_pipeline.py | 36 +++- paper/paper.md | 61 +++--- src/saxsabs/cli.py | 174 +++++++++++++++--- src/saxsabs/core/buffer_subtraction.py | 19 ++ src/saxsabs/core/intensity_state.py | 14 +- src/saxsabs/io/__init__.py | 4 + src/saxsabs/io/parsers.py | 117 ++++++++---- src/saxsabs/io/writers.py | 37 +++- tests/test_buffer_subtraction.py | 52 ++++-- tests/test_cli.py | 160 +++++++++++++++- tests/test_intensity_state.py | 10 + tests/test_io_formats.py | 104 ++++++++--- tests/test_parsers.py | 12 +- tests/test_workbench_scientific.py | 24 ++- 27 files changed, 849 insertions(+), 225 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9994aa8..82f2926 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,24 @@ ## [2.0.0] - Unreleased +### JOSS reviewer polish (2026-08-16) + +- Make `saxsabs estimate-k --ref` optional so the CLI can use the same built-in + NIST SRM 3600 curve as the Python API; reject `--ref-q-col`/`--ref-i-col` + without `--ref`. +- Gate CLI `estimate-k` and `subtract-buffer` (and the `minimal_2d` example) + through the intensity-state contract; refuse unlabeled intensity before $K$. +- Add `estimate-k --thickness-cm` (centimetres; Workbench Tab 1 remains mm) and + a `subtract-buffer` subcommand. +- Stamp `1/cm` only when writers receive `absolute_cm^-1` plus an explicit + cm$^{-1}$ unit; readers expose `i_abs`/`i_rel` only for those assessed states. + A unitless metadata label `absolute` is ambiguous, not cm$^{-1}$. +- Point README and paper at the unreleased 2.0.0 tree on `main`, not GitHub + Release v1.1.1. Record canSAS XSD / punx checks as offline 2026-08-15, not CI. +- Tighten the paper's K-estimator, interface-equivalence, and `minimal_2d` + claims, and remove stale Draft-PR #1 merge instructions from the README and + readiness notes. + ### Submission-candidate hardening (2026-08-12) - Require finite positive inherited `thickness_cm` and a non-empty diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 42090aa..aeeee76 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -20,7 +20,7 @@ git clone https://github.com/D-sudoasd/SASAbs.git cd SASAbs python -m pip install -e ".[dev]" pytest -q -ruff check src tests +ruff check SASAbs.py saxs_mpl_style.py src tests paper/*.py scripts/*.py ``` Install `.[gui]`, `.[hdf5]`, `.[io]`, or `.[bl19b2]` only when the change needs @@ -33,7 +33,9 @@ Keep pull requests focused. For a behavior change: - add or update focused tests; - keep reusable scientific logic in `src/saxsabs/` and GUI orchestration separate; - update public CLI/API documentation when its behavior changes; -- run `pytest -q` and `ruff check src tests` locally; +- run `pytest -q` and + `ruff check SASAbs.py saxs_mpl_style.py src tests paper/*.py scripts/*.py` + locally; - describe the workflow, validation performed, and any remaining limitations. Maintainers review pull requests for scientific input semantics, provenance, diff --git a/README.md b/README.md index 9c40812..e7b9747 100644 --- a/README.md +++ b/README.md @@ -20,6 +20,12 @@ reusable data writers, and provenance checks. The result and the processing record remain reviewable together. +Reviewers should use the unreleased 2.0.0 tree on +[`main`](https://github.com/D-sudoasd/SASAbs). GitHub Release +[v1.1.1](https://github.com/D-sudoasd/SASAbs/releases/tag/v1.1.1) is an earlier +archive and is not this candidate. Do not treat the Zenodo concept DOI as a +version DOI for 2.0.0. +

Quick start · Choose a workflow · @@ -77,7 +83,7 @@ a display server. | Route | Best for | Start here | | --- | --- | --- | -| **CLI utilities** | normalization, header and 1D parsing, robust K estimation | `saxsabs --help` | +| **CLI utilities** | normalization, header and 1D parsing, gated K estimation, gated buffer subtraction | `saxsabs --help` | | **SAXSAbs Workbench** | interactive K calibration, batch processing, external-1D scaling | `saxsabs-workbench --lang en` | | **Strict BL19B2 runner** | validated campaign inputs under current BL19B2 conventions | [batch runbook](docs/bl19b2_abs2d_batch_runbook.md) | | **Python API** | reusable scientific calculations and file I/O | [API reference](docs/api.md) | @@ -124,8 +130,10 @@ python examples/minimal_2d/run_minimal_2d_pipeline.py ``` It writes inspectable CSV, TSV, and XML outputs, plus HDF5 when `h5py` is -installed. The acceptance summary requires `k_relative_error < 0.005` and -`sample_max_relative_error < 0.01`. See the +installed. The script gates the standard profile as `relative` before $K$, +writes `absolute_cm^-1` metadata, and checks that the XML exposes `i_abs` +rather than `i_rel`. The acceptance summary requires `k_relative_error < 0.005` +and `sample_max_relative_error < 0.01`. See the [example documentation](examples/minimal_2d/README.md) for construction details and expected files. @@ -133,8 +141,9 @@ and expected files. Deterministic synthetic K-factor example showing retained and rejected ratios.

-> This example checks software arithmetic and generated file content. It is not -> measured-beamline validation or independent third-party format validation. +> This example recovers a planted synthetic $K$ and sample curve and checks +> labeled file content. It is not measured-beamline validation or independent +> third-party format validation. ## Documentation @@ -151,10 +160,12 @@ and expected files. Absolute calibration depends on a suitable reference, detector geometry, monitor semantics, transmission, thickness, and instrument-specific provenance. -The strict 2D workflow currently targets BL19B2 conventions. canSAS1d output is -checked against the official version 1.1 XSD. NXcanSAS output passes project-local -round-trip tests and punx 0.3.5 with its bundled v2018.5 definitions; current -NeXus definitions and third-party consumers have not yet been verified. +The strict 2D workflow currently targets BL19B2 conventions. canSAS1d and +NXcanSAS layouts are covered by project-local round-trip tests. An offline +check on 15 August 2026 validated the deterministic example against the official +canSAS1d 1.1 XSD and punx 0.3.5 with its bundled v2018.5 definitions; that check +is not in CI. Current NeXus definitions and third-party consumers have not been +verified. ## Development @@ -176,9 +187,9 @@ python scripts/check_submission_readiness.py \ ``` Run the strict command from the exact branch and commit that will be submitted. -If Draft PR #1 is merged first, check out the resulting clean `main`, update -`submitted_branch` and `submitted_commit` in the confirmation JSON, and rerun -the gate. A PASS recorded before the merge does not cover the merge commit. +PR #1 is already on `main`. A PASS recorded on an earlier revision does not +cover a later commit; update `submitted_branch` and `submitted_commit` and rerun +the gate on the revision sent to JOSS. After the strict local gate passes, verify the same commit, branch, visible README and paper, repository identity, and successful CI run against GitHub: diff --git a/SASAbs.py b/SASAbs.py index 40e8c30..22ed9e0 100644 --- a/SASAbs.py +++ b/SASAbs.py @@ -1340,10 +1340,34 @@ def should_skip_all_existing(existing_flags, policy): write_nxcansas_h5 = None try: - from saxsabs.io.parsers import read_external_1d_profile as _core_read_external_1d_profile + from saxsabs.io.parsers import ( + profile_intensity as _core_profile_intensity, + profile_uncertainty as _core_profile_uncertainty, + read_external_1d_profile as _core_read_external_1d_profile, + ) except Exception: + _core_profile_intensity = None + _core_profile_uncertainty = None _core_read_external_1d_profile = None + +def _profile_intensity(profile): + if _core_profile_intensity is not None: + return _core_profile_intensity(profile) + for key in ("intensity", "i_abs", "i_rel"): + if key in profile: + return np.asarray(profile[key], dtype=np.float64) + raise KeyError("profile has no intensity array") + + +def _profile_uncertainty(profile): + if _core_profile_uncertainty is not None: + return _core_profile_uncertainty(profile) + for key in ("uncertainty", "err_abs", "err_rel"): + if key in profile: + return np.asarray(profile[key], dtype=np.float64) + raise KeyError("profile has no uncertainty array") + try: from saxsabs.io.calibrated2d import ( SCHEMA_VERSION as CALIBRATED2D_SCHEMA_VERSION, @@ -5657,18 +5681,29 @@ def subtract_external_absolute_buffer( if subtract_buffer is None: raise RuntimeError("formal buffer subtraction kernel is unavailable") + sample_profile = { + "intensity_state": "absolute_cm^-1", + "intensity_unit": "1/cm", + "i_col": "I_abs_cm^-1", + "operator_provenance": { + "intensity_state": "absolute_cm^-1", + "corrections_applied": '["k","thickness"]', + }, + } args = ( np.asarray(sample_q, dtype=np.float64), np.asarray(sample_i, dtype=np.float64), np.asarray(sample_err, dtype=np.float64), np.asarray(buffer_profile["x"], dtype=np.float64), - np.asarray(buffer_profile["i_rel"], dtype=np.float64), - np.asarray(buffer_profile["err_rel"], dtype=np.float64), + _profile_intensity(buffer_profile), + _profile_uncertainty(buffer_profile), ) result = subtract_buffer( *args, alpha=alpha, alpha_uncertainty=alpha_uncertainty, + sample_profile=sample_profile, + buffer_profile=buffer_profile, ) if result.err_statistical is None: raise RuntimeError("buffer kernel did not return statistical uncertainty") @@ -6134,8 +6169,11 @@ def align_profile_to_x(self, x_target, ref_profile, name): xr, yr, er = self._regularize_xy_triplet( ref_profile["x"], - ref_profile["i_rel"], - ref_profile.get("err_rel"), + _profile_intensity(ref_profile), + ref_profile.get( + "uncertainty", + ref_profile.get("err_abs", ref_profile.get("err_rel")), + ), min_points=2, name=name, ) @@ -6713,8 +6751,8 @@ def run_external_1d_batch(self): else: thk_cm_used = fixed_thk_cm thickness_source = "tab3_fixed_thickness_input" - i_abs = np.asarray(prof["i_rel"], dtype=np.float64) * scale_factor - err_abs = np.asarray(prof["err_rel"], dtype=np.float64) * abs(scale_factor) + i_abs = _profile_intensity(prof) * scale_factor + err_abs = _profile_uncertainty(prof) * abs(scale_factor) else: sp = self.resolve_external_sample_params(fp, meta_map, monitor_mode) norm_s = sp["norm"] @@ -6742,8 +6780,8 @@ def run_external_1d_batch(self): thickness_source = str(provenance["thickness_source"]) scale_factor = k - s_i = np.asarray(prof["i_rel"], dtype=np.float64) - s_e = np.asarray(prof["err_rel"], dtype=np.float64) + s_i = _profile_intensity(prof) + s_e = _profile_uncertainty(prof) x = np.asarray(prof["x"], dtype=np.float64) self.assert_external_profile_axis_compatible(prof, bg_prof, "BG") @@ -10867,7 +10905,7 @@ def _get_std_reference_data(self): from saxsabs.io.parsers import read_external_1d_profile prof = read_external_1d_profile(ref_path) q_user = prof["x"] - i_user = prof["i_rel"] + i_user = _profile_intensity(prof) return get_reference_data(key, q_user=q_user, i_user=i_user) elif key == "Water_20C": temp_c = self.t1_water_temp.get() diff --git a/SUBMISSION_READINESS.md b/SUBMISSION_READINESS.md index 2e7e0cc..be181c0 100644 --- a/SUBMISSION_READINESS.md +++ b/SUBMISSION_READINESS.md @@ -2,6 +2,9 @@ Updated: 16 August 2026 (Asia/Shanghai) +Review the unreleased 2.0.0 tree on `main`, not GitHub Release v1.1.1. Do not +create `v2.0.0`, a GitHub Release, or a Zenodo version archive during review. + ## Locally verified - Full source suite: PASS in a fully provisioned Python 3.13 environment; exact @@ -30,10 +33,11 @@ Updated: 16 August 2026 (Asia/Shanghai) have green push and Draft-PR runs for the complete matrix. Immutable commit IDs and run URLs belong in the dated external validation record rather than this tracked file, because editing the evidence here creates a new HEAD. -- External format checks: the deterministic example's canSAS1d XML validates - against the official 1.1 XSD with zero errors. Its NXcanSAS HDF5 output passes - punx 0.3.5 with bundled v2018.5 definitions (97 OK, 0 WARN, 0 ERROR); current - NeXus definitions and third-party consumers remain unverified. +- External format checks (offline 15 August 2026, not in CI): the deterministic + example's canSAS1d XML validated against the official 1.1 XSD with zero + errors. Its NXcanSAS HDF5 output passed punx 0.3.5 with bundled v2018.5 + definitions (97 OK, 0 WARN, 0 ERROR); current NeXus definitions and + third-party consumers remain unverified. ## Must be resolved before submission @@ -59,10 +63,10 @@ python scripts/check_submission_readiness.py \ --manual-confirmations path/to/submission-confirmations.json ``` -The gate must run on the exact branch and commit submitted to JOSS. If the -Draft PR is merged first, rerun it on the clean resulting `main` commit and -record `submitted_branch` and `submitted_commit` accordingly; pre-merge -evidence is not evidence for a later merge commit. +The gate must run on the exact branch and commit submitted to JOSS. PR #1 is +already on `main`; rerun the gate on the clean `main` commit that will be +submitted and record `submitted_branch` and `submitted_commit` accordingly. +Evidence from an earlier revision is not evidence for a later commit. After that local PASS, run: diff --git a/docs/api.md b/docs/api.md index fefcd9e..0df9820 100644 --- a/docs/api.md +++ b/docs/api.md @@ -43,8 +43,11 @@ estimate_k_factor_robust( Interpolates the measured profile on the reference q grid in `q_window`, forms `I_ref / I_meas`, and applies median/MAD outlier rejection. If both reference -arrays are omitted, it uses the built-in NIST SRM 3600 reference. The result -contains the estimate and diagnostics; inspect it before applying a scale. +arrays are omitted, it uses the built-in NIST SRM 3600 reference and fails +closed if any ratio exceeds the certificate-derived relative-intensity limit +before that filter. Measured intensities must already be on the same scale as +the reference (cm$^{-1}$ for SRM 3600). The result contains the estimate and +diagnostics; inspect it before applying a scale. ```python from saxsabs import estimate_k_factor_robust @@ -87,7 +90,9 @@ print(mu.mu_linear_cm_inv) assess_intensity_state(profile: Mapping[str, object]) -> IntensityStateAssessment subtract_buffer(q_sample, i_sample, err_sample, q_buffer, i_buffer, err_buffer, alpha: float = 1.0, high_q_diag: tuple[float, float] = (0.15, 0.25), - *, alpha_uncertainty: float | None = None) -> BufferSubtractionResult + *, alpha_uncertainty: float | None = None, + sample_profile: Mapping[str, object], + buffer_profile: Mapping[str, object]) -> BufferSubtractionResult propagate_absolute_uncertainty(intensity: np.ndarray, *, statistical_standard_uncertainty=None, k_relative_standard_uncertainty=None, standard_relative_standard_uncertainty=None, @@ -99,8 +104,10 @@ propagate_absolute_uncertainty(intensity: np.ndarray, *, ``` `assess_intensity_state` classifies metadata, units, and correction information; -conflicting evidence remains ambiguous. `subtract_buffer` interpolates a buffer -onto the sample q grid when necessary and propagates supplied uncertainties. +conflicting evidence remains ambiguous. A unitless metadata label `absolute` is +ambiguous and is not treated as cm$^{-1}$. `subtract_buffer` requires +`sample_profile` and `buffer_profile` provenance, interpolates a buffer onto +the sample q grid when necessary, and propagates supplied uncertainties. `propagate_absolute_uncertainty` combines statistical and supplied standard uncertainty components; relative inputs must be relative standard uncertainties. @@ -117,20 +124,21 @@ write_nxcansas_h5(path: str | Path, q: np.ndarray, i_abs: np.ndarray, ``` `read_external_1d_profile` reads supported text/tabular profiles and routes XML -and HDF5 extensions to canSAS/NXcanSAS readers. The returned mapping includes -the parsed arrays and parsing metadata. Writers expect q in Å⁻¹, absolute -intensity in cm⁻¹, and optional uncertainty in cm⁻¹. NXcanSAS writing requires -the optional `h5py` dependency. +and HDF5 extensions to canSAS/NXcanSAS readers. The returned mapping always +includes `intensity` and `uncertainty`. It adds `i_abs`/`err_abs` only for +assessed `absolute_cm^-1` profiles and `i_rel`/`err_rel` only for assessed +`relative` profiles. Writers expect q in Å⁻¹ and stamp `1/cm` only when +metadata declares `intensity_state=absolute_cm^-1` and an explicit cm⁻¹ unit. +NXcanSAS writing requires the optional `h5py` dependency. ```python -from saxsabs import read_external_1d_profile +from saxsabs.io import profile_intensity, read_external_1d_profile profile = read_external_1d_profile("examples/profile_example.csv") -print(profile["x"], profile["i_rel"]) +print(profile["x"], profile_intensity(profile), profile.get("intensity_state")) ``` -The parser deliberately names an uncalibrated intensity array `i_rel`. Do not -pass that array to an absolute-intensity writer. Call `write_cansas1d_xml` or +Do not assume `i_rel` is present. Call `write_cansas1d_xml` or `write_nxcansas_h5` only after a validated calibration has established the absolute intensity, uncertainty, units, and provenance. @@ -149,8 +157,11 @@ for parameters. Required inputs are shown in angle brackets: saxsabs norm-factor --mon --trans <0 --mode [--exp ] saxsabs parse-header --header-json saxsabs parse-external1d --input -saxsabs estimate-k --meas --ref [--q-col ] [--i-col ] +saxsabs estimate-k --meas [--ref ] [--q-col ] [--i-col ] [--ref-q-col ] [--ref-i-col ] [--qmin ] [--qmax ] + [--intensity-state relative] [--thickness-cm ] +saxsabs subtract-buffer --sample --buffer [--alpha ] + [--alpha-uncertainty ] saxsabs bl19b2-abs2d --input-root (--poni |--pydidas-cali-yaml ) (--mu |--sample-thickness-cm ) --monitor-mode [workflow options] @@ -165,7 +176,8 @@ The main commands are: | `norm-factor` | exposure, monitor, transmission, mode | normalization value | | `parse-header` | header JSON | extracted exposure/monitor/transmission JSON | | `parse-external1d` | profile path | parsed-profile summary JSON | -| `estimate-k` | measured and reference tabular profiles | K-factor result JSON | +| `estimate-k` | relative measured profile; optional reference (built-in SRM 3600 if omitted); optional `--thickness-cm` | K-factor result JSON | +| `subtract-buffer` | absolute sample and buffer profiles with cm⁻¹ units | subtraction diagnostic JSON | | `bl19b2-abs2d` | explicit BL19B2 inputs and semantics | batch result JSON and requested files | | `bl19b2-abs2d-v1-legacy` | explicit migration choices | legacy-compatible batch result with documented assumptions | @@ -179,4 +191,5 @@ These APIs provide software operations and checks; they do not establish that a beamline measurement is calibrated. Users remain responsible for appropriate reference standards, independently measured inputs, detector geometry, valid units, and experiment-specific acceptance. The included synthetic 2D example -tests arithmetic and interoperability, not experimental beamline validation. +recovers a planted $K$ and sample curve and checks labeled file content; it is +not experimental beamline validation. diff --git a/docs/architecture.md b/docs/architecture.md index 27ef62e..f7321ab 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -35,11 +35,13 @@ deterministic and testable. - **`src/saxsabs/io`**: robust input parsing plus standard-format writers (canSAS XML and NXcanSAS HDF5). -- **`src/saxsabs/cli.py`**: six headless subcommands: four focused utilities - (`norm-factor`, `parse-header`, `parse-external1d`, `estimate-k`), the - safety-first `bl19b2-abs2d` workflow, and the explicit - `bl19b2-abs2d-v1-legacy` migration entry. The legacy entry requires explicit - monitor and thickness semantics and never silently restores v1 defaults. +- **`src/saxsabs/cli.py`**: seven headless subcommands: five focused utilities + (`norm-factor`, `parse-header`, `parse-external1d`, `estimate-k`, + `subtract-buffer`), the safety-first `bl19b2-abs2d` workflow, and the + explicit `bl19b2-abs2d-v1-legacy` migration entry. `estimate-k` and + `subtract-buffer` apply the intensity-state gates; the first three utilities + remain thin parsers/calculators. The legacy entry requires explicit monitor + and thickness semantics and never silently restores v1 defaults. - **`src/saxsabs/constants.py`**: pluggable reference-standard registry (SRM 3600, water, custom curves). - **`src/saxsabs/workbench_launcher.py`**: packaged launcher used by diff --git a/docs/joss-submission-checklist.md b/docs/joss-submission-checklist.md index 538e44c..f7fa960 100644 --- a/docs/joss-submission-checklist.md +++ b/docs/joss-submission-checklist.md @@ -64,11 +64,13 @@ accessed 16 August 2026: table and does not describe either as XCOM. - [x] The workflow figure has editable SVG/PDF sources and the GUI image is a window-scoped capture of the actual Workbench. -- [x] canSAS1d XML from the deterministic example validates against the official - version 1.1 XSD with zero errors. -- [ ] NXcanSAS HDF5 passes punx 0.3.5 with its bundled v2018.5 definitions, but - current NeXus definitions and a third-party application consumer remain - unverified because punx 0.3.5 cannot parse the current definition set. +- [x] canSAS1d XML from the deterministic example validated offline on + 15 August 2026 against the official version 1.1 XSD with zero errors. + That check is not in CI. +- [ ] NXcanSAS HDF5 passed punx 0.3.5 offline on 15 August 2026 with its + bundled v2018.5 definitions; that check is not in CI. Current NeXus + definitions and a third-party application consumer remain unverified + because punx 0.3.5 cannot parse the current definition set. - [ ] The author confirms author order, affiliation, corresponding author, acknowledgements, funding, conflicts of interest, and contribution roles. - [ ] The author confirms the complete AI disclosure and human review statement. diff --git a/docs/reviewer-faq.md b/docs/reviewer-faq.md index 0fd5a52..2187d43 100644 --- a/docs/reviewer-faq.md +++ b/docs/reviewer-faq.md @@ -26,19 +26,29 @@ python examples/minimal_2d/run_minimal_2d_pipeline.py ``` The script constructs independent dark, blank, SRM 3600, and sample frames, -produces deterministic outputs (CSV/TSV/canSAS XML and optional NXcanSAS HDF5), -and writes numerical K and sample-intensity errors to `summary.json`. This is a -software golden test, not a substitute for measured beamline validation. +gates the standard profile as `relative` before $K$, writes labeled +`absolute_cm^-1` outputs (CSV/TSV/canSAS XML and optional NXcanSAS HDF5), +checks that the XML exposes `i_abs` not `i_rel`, and writes numerical K and +sample-intensity errors to `summary.json`. This recovers a planted synthetic +curve within script tolerances. It is a software golden test, not measured +beamline validation or third-party format validation. ## Have the structured exports been checked outside the project readers? -Yes, with a bounded result. On 15 August 2026, the minimal example's XML output -validated with zero errors against `cansas1d.xsd` from the official canSAS -`1dwg` repository (blob `c376e590bf6c297ee5664834183b6d09b5684318`). -The HDF5 output passed punx 0.3.5 against its bundled NeXus v2018.5 definitions -with 97 OK, 0 WARN, and 0 ERROR findings. punx 0.3.5 could not parse the current -NeXus `main` definitions (commit `6313522`), so the project does not claim -validation against current definitions or a third-party application consumer. +Yes, with a bounded offline result that is not in CI. On 15 August 2026, the +minimal example's XML output validated with zero errors against `cansas1d.xsd` +from the official canSAS `1dwg` repository (blob +`c376e590bf6c297ee5664834183b6d09b5684318`). The HDF5 output passed punx 0.3.5 +against its bundled NeXus v2018.5 definitions with 97 OK, 0 WARN, and 0 ERROR +findings. punx 0.3.5 could not parse the current NeXus `main` definitions +(commit `6313522`), so the project does not claim validation against current +definitions or a third-party application consumer. + +## Which revision should be reviewed? + +Review the unreleased 2.0.0 tree on `main`. GitHub Release v1.1.1 is an earlier +archive and is not this candidate. Do not create `v2.0.0` or a new Zenodo +version record during review. ## What is the software boundary? diff --git a/examples/k_measured.csv b/examples/k_measured.csv index 76bff51..72058ea 100644 --- a/examples/k_measured.csv +++ b/examples/k_measured.csv @@ -1,3 +1,4 @@ +# intensity_state: relative q,i 0.01,17.1 0.02,15.4 diff --git a/examples/k_reference.csv b/examples/k_reference.csv index 12e1c3f..76e3b32 100644 --- a/examples/k_reference.csv +++ b/examples/k_reference.csv @@ -1,3 +1,5 @@ +# intensity_state: absolute_cm^-1 +# intensity_unit: 1/cm q,i 0.01,34.2 0.02,30.8 diff --git a/examples/manual-verification.md b/examples/manual-verification.md index 69cb661..3a186a5 100644 --- a/examples/manual-verification.md +++ b/examples/manual-verification.md @@ -53,6 +53,10 @@ cannot be fully public. saxsabs estimate-k --meas examples/k_measured.csv --ref examples/k_reference.csv --qmin 0.01 --qmax 0.2 ``` + The measured file already declares `# intensity_state: relative`. Unlabeled + intensity is refused. Thickness, when needed, is `--thickness-cm` (Workbench + Tab 1 enters millimetres: `1.055 mm = 0.1055 cm`). + Expected: `k_factor` close to `2.0`, with non-zero `points_used`. 7. Verify the deterministic synthetic raw-frame → absolute intensity workflow: diff --git a/examples/minimal_2d/README.md b/examples/minimal_2d/README.md index 3664f61..06aa120 100644 --- a/examples/minimal_2d/README.md +++ b/examples/minimal_2d/README.md @@ -26,6 +26,7 @@ Expected key result: - `absolute_profile.csv`, `absolute_profile.tsv`, `absolute_profile.xml` - `absolute_profile.h5` if `h5py` is installed (`pip install -e .[hdf5]`) -The example validates software arithmetic and file interoperability. It does not -replace validation with a measured SRM 3600 coupon and independently measured -transmissions at the target beamline. +The example recovers a planted synthetic $K$ and sample curve, writes labeled +`absolute_cm^-1` outputs, and checks that the XML exposes `i_abs` rather than +`i_rel`. It does not replace validation with a measured SRM 3600 coupon, +independently measured transmissions, or a third-party format consumer. diff --git a/examples/minimal_2d/run_minimal_2d_pipeline.py b/examples/minimal_2d/run_minimal_2d_pipeline.py index a2d6bfa..3462df1 100644 --- a/examples/minimal_2d/run_minimal_2d_pipeline.py +++ b/examples/minimal_2d/run_minimal_2d_pipeline.py @@ -17,9 +17,11 @@ build_nist_net_image, estimate_k_factor_robust, get_reference_data, + require_relative_input_for_absolute_scaling, write_cansas1d_xml, write_nxcansas_h5, ) +from saxsabs.io.parsers import read_cansas1d_xml K_TRUE = 2.0 @@ -152,9 +154,24 @@ def run_pipeline(output_dir: Path) -> dict[str, object]: _, sample_expected = radial_average(sample_absolute_image, center) q_ref, i_ref = get_reference_data("SRM3600") + standard_profile = { + "x": q, + "intensity": standard_measured / SRM3600_THICKNESS_CM, + "i_col": "i_standard_measured_per_cm", + "intensity_state": "relative", + "operator_provenance": { + "intensity_state": "relative", + "corrections_applied": ( + '["background","dark","monitor","thickness","transmission"]' + ), + }, + } + require_relative_input_for_absolute_scaling( + standard_profile, profile_name="minimal_2d standard" + ) k_result = estimate_k_factor_robust( q_meas=q, - i_meas_per_cm=standard_measured / SRM3600_THICKNESS_CM, + i_meas_per_cm=standard_profile["intensity"], q_ref=q_ref, i_ref=i_ref, q_window=(max(0.01, float(q.min())), float(q.max())), @@ -196,6 +213,11 @@ def run_pipeline(output_dir: Path) -> dict[str, object]: "detector_name": "synthetic-array", "process_name": "minimal_2d_pipeline", "uncertainty_status": "unknown_without_input_variances", + "intensity_state": "absolute_cm^-1", + "intensity_unit": "1/cm", + "corrections_applied": ( + '["background","dark","k","monitor","thickness","transmission"]' + ), } write_cansas1d_xml( output_dir / "absolute_profile.xml", @@ -217,8 +239,20 @@ def run_pipeline(output_dir: Path) -> dict[str, object]: except ImportError: pass + written_xml = read_cansas1d_xml(output_dir / "absolute_profile.xml") + if written_xml.get("intensity_state") != "absolute_cm^-1": + raise RuntimeError("written XML is not labeled absolute_cm^-1") + if "i_rel" in written_xml: + raise RuntimeError("absolute XML must not expose an i_rel label") + if "i_abs" not in written_xml: + raise RuntimeError("absolute XML must expose i_abs") + summary: dict[str, object] = { "validation_type": "deterministic_synthetic_raw_frames", + "evidence": ( + "planted synthetic K and sample curve recovered within script tolerances; " + "not measured-beamline or third-party format validation" + ), "points": int(q.size), "q_min": float(q.min()), "q_max": float(q.max()), diff --git a/paper/paper.md b/paper/paper.md index 20ae261..b158ee1 100644 --- a/paper/paper.md +++ b/paper/paper.md @@ -71,11 +71,12 @@ for external 1D data and the current BL19B2 2D workflow. It builds on pyFAI and FabIO rather than reimplementing detector integration and image access. The scholarly contribution is the explicit intensity-state and correction-history contract across calibration, external-profile scaling, and traceable export--a -boundary not provided by those dependencies. Its Python, command-line, and -graphical interfaces reuse core checks for physical inputs, processing history, -and output state, although the Workbench is not an equivalent front end to the -strict campaign runner. Geometry calibration and model fitting remain with the -specialist tools above. +boundary not provided by those dependencies. The Python API, command line, and +Workbench share those numerical and I/O modules, but they are not equivalent +front ends. CLI `estimate-k` and `subtract-buffer` apply the intensity-state +gates; `norm-factor`, `parse-header`, and `parse-external1d` remain thin +utilities. The Workbench is not a substitute for the strict campaign runner. +Geometry calibration and model fitting remain with the specialist tools above. # Software design @@ -87,9 +88,9 @@ parses heterogeneous headers and 1D tables and writes canSAS1d 1.1 XML and NXcanSAS 1.1 HDF5 [@cansas1d; @nxcansas]. The strict BL19B2 workflow validates detector, monitor, transmission, thickness, reference, and output inputs before integration, calibration, and export. CLI subcommands cover normalization, -parsing, and $K$ estimation; the SAXSAbs Workbench adds interactive calibration, -batch processing, and external-1D conversion -(\autoref{fig:gui}). +parsing, gated $K$ estimation, and gated buffer subtraction; the SAXSAbs +Workbench adds interactive calibration, batch processing, and external-1D +conversion (\autoref{fig:gui}). The design deliberately separates a bounded, strict BL19B2 campaign schema from the more general calculation and I/O APIs. A permissive all-beamline workflow @@ -105,11 +106,13 @@ reference grid and calculates $R_i=I_{\mathrm{ref}}(q_i)/I_{\mathrm{meas}}(q_i)$. It defines the median ratio as $\tilde{R}$ and $\hat{\sigma}=1.4826\,\mathrm{median}(|R_i-\tilde{R}|)$, retains ratios within -$3\hat{\sigma}$, and uses their median as $K$. This rule excludes isolated -anomalous ratios. The software reports the dispersion of retained ratios -separately from combined calibration uncertainty and propagates supported -independent input uncertainties when supplied; unavailable terms remain -unspecified. The BL19B2 workflow reports a partial combined standard uncertainty +$3\hat{\sigma}$, and uses their median as $K$. Isolated anomalous ratios are +excluded by this filter on user-supplied references; the built-in SRM 3600 path +additionally fails closed if any ratio exceeds the certificate-derived +relative-intensity limit before filtering. The software reports the dispersion +of retained ratios separately from combined calibration uncertainty and +propagates supported independent input uncertainties when supplied; unavailable +terms remain unspecified. The BL19B2 workflow reports a partial combined standard uncertainty when shared covariance terms are not quantified and does not report a system expanded uncertainty in that case. @@ -117,12 +120,15 @@ The attenuation functionality follows two distinct data paths. Its general diagnostic calculator obtains energy-dependent elemental coefficients from the Elam database through `xraydb.mu_elam` [@elam2002; @xraydb]. Its fixed 30 keV material calculation uses a versioned NIST SRD 126 snapshot [@nist_srd126]. -Absolute 1D intensity is reported in cm$^{-1}$. CSV and TSV outputs are directly +Absolute 1D intensity is reported in cm$^{-1}$ only when the writer receives +`intensity_state=absolute_cm^-1` and an explicit cm$^{-1}$ unit; a unitless +`absolute` label is not treated as cm$^{-1}$. CSV and TSV outputs are directly inspectable; the structured XML and HDF5 outputs follow the documented canSAS1d -1.1 and NXcanSAS 1.1 layouts. The XML output validates against the official -canSAS1d 1.1 XSD. NXcanSAS output passes `punx` 0.3.5 [@punx] with its bundled -v2018.5 definitions, but current NeXus definitions and third-party consumers -have not yet been verified. +1.1 and NXcanSAS 1.1 layouts. Project-local tests cover those layouts. An +offline check on 15 August 2026 validated the deterministic example against the +official canSAS1d 1.1 XSD and `punx` 0.3.5 [@punx] with bundled v2018.5 +definitions; that check is not in CI, and current NeXus definitions and +third-party consumers have not been verified. ![SAXSAbs Workbench in English, showing K-calibration inputs and the plotting area. The screenshot was captured from the current source tree and contains no beamline data.](fig_gui.png){#fig:gui width="100%"} @@ -132,17 +138,22 @@ Source code, tests, documentation, and examples are available in the [SASAbs GitHub repository](https://github.com/D-sudoasd/SASAbs) under the BSD-3-Clause license. The core package supports Python 3.10 and later; optional dependency groups enable Workbench, detector-image, BL19B2, and HDF5 functionality. The -README includes installation instructions and minimal commands. Archived -releases are collected in the Zenodo concept record [@saxsabs_archive]. +README includes installation instructions and minimal commands. Reviewers +should use the unreleased 2.0.0 tree on `main`, not GitHub Release v1.1.1. +Archived earlier releases are collected in the Zenodo concept record +[@saxsabs_archive]. # Research impact statement The repository includes a strict BL19B2 batch workflow from detector images to -exported results. Its deterministic example in `examples/minimal_2d/` generates -synthetic dark, background, standard, and sample images; recovers the specified -calibration factor within the tested tolerance; and writes text, canSAS, and -NXcanSAS outputs. Automated tests cover numerical calculations, parsers, -exporters, the command-line interface, launchers, and Workbench validation rules. +exported results. Its deterministic example in `examples/minimal_2d/` plants +synthetic dark, background, standard, and sample frames; recovers the planted +$K$ and sample curve within script tolerances; writes labeled absolute text and +canSAS XML (and NXcanSAS when `h5py` is present); and checks that the XML +exposes `i_abs`, not `i_rel`. That example is a software golden test, not +measured-beamline or third-party format validation. Automated tests cover +numerical calculations, parsers, exporters, the command-line interface, +launchers, and Workbench validation rules. The repository configures continuous integration for Python 3.10--3.13 on Linux, Windows, and macOS. diff --git a/src/saxsabs/cli.py b/src/saxsabs/cli.py index 7b52dd6..5cdb051 100644 --- a/src/saxsabs/cli.py +++ b/src/saxsabs/cli.py @@ -1,6 +1,6 @@ """Command-line interface for headless SAXS calibration operations. -Provides six subcommands: four small utilities plus the safe BL19B2 workflow +Provides seven subcommands: five small utilities plus the safe BL19B2 workflow and its explicit v1 migration entry. """ @@ -15,9 +15,16 @@ import pandas as pd from . import __version__ -from .core.normalization import compute_norm_factor +from .core.buffer_subtraction import subtract_buffer from .core.calibration import estimate_k_factor_robust -from .io.parsers import parse_header_values, read_external_1d_profile +from .core.intensity_state import require_relative_input_for_absolute_scaling +from .core.normalization import compute_norm_factor +from .io.parsers import ( + parse_header_values, + profile_intensity, + profile_uncertainty, + read_external_1d_profile, +) def _die(message: str) -> None: @@ -110,10 +117,9 @@ def _read_profile_for_estimate( q_col: str | None, i_col: str | None, profile_label: str, -) -> tuple[object, object]: +) -> dict[str, object]: if q_col is None and i_col is None: - parsed = read_external_1d_profile(path) - return parsed["x"], parsed["i_rel"] + return read_external_1d_profile(path) df = _read_tabular_dataframe(path) columns = list(df.columns) @@ -122,7 +128,26 @@ def _read_profile_for_estimate( q = pd.to_numeric(df[resolved_q_col], errors="coerce").to_numpy(dtype=float) intensity = pd.to_numeric(df[resolved_i_col], errors="coerce").to_numpy(dtype=float) - return q, intensity + profile = read_external_1d_profile(path) + profile["x"] = q + profile["intensity"] = intensity + profile["i_col"] = str(resolved_i_col) + profile["x_col"] = str(resolved_q_col) + return profile + + +def _apply_declared_intensity_state( + profile: dict[str, object], + declared_state: str | None, +) -> dict[str, object]: + if not declared_state: + return profile + provenance = dict(profile.get("operator_provenance") or {}) + provenance["intensity_state"] = declared_state + updated = dict(profile) + updated["operator_provenance"] = provenance + updated["intensity_state"] = declared_state + return updated def _add_bl19b2_arguments( @@ -315,14 +340,49 @@ def build_parser() -> argparse.ArgumentParser: p_profile.add_argument("--input", required=True, type=Path) p_k = sub.add_parser("estimate-k", help="Estimate robust K-factor from measured and reference CSV") - p_k.add_argument("--meas", required=True, type=Path, help="Measured profile CSV with columns q,i") - p_k.add_argument("--ref", required=True, type=Path, help="Reference profile CSV with columns q,i") + p_k.add_argument( + "--meas", + required=True, + type=Path, + help="Measured profile on the same intensity scale as the reference (cm^-1 for SRM 3600)", + ) + p_k.add_argument( + "--ref", + default=None, + type=Path, + help="Reference profile CSV; omit to use the built-in NIST SRM 3600 curve", + ) p_k.add_argument("--q-col", default=None, help="Measured q column override") p_k.add_argument("--i-col", default=None, help="Measured intensity column override") p_k.add_argument("--ref-q-col", default=None, help="Reference q column override") p_k.add_argument("--ref-i-col", default=None, help="Reference intensity column override") p_k.add_argument("--qmin", type=float, default=0.01) p_k.add_argument("--qmax", type=float, default=0.2) + p_k.add_argument( + "--intensity-state", + default=None, + help="Declare measured intensity_state when the file has no provenance " + "(relative required for K estimation)", + ) + p_k.add_argument( + "--thickness-cm", + type=float, + default=None, + help=( + "Standard thickness in cm used to convert measured intensity to cm^-1 " + "before K estimation. Workbench Tab 1 enters millimetres " + "(1.055 mm = 0.1055 cm)." + ), + ) + + p_sub = sub.add_parser( + "subtract-buffer", + help="Subtract an absolute buffer from an absolute sample profile", + ) + p_sub.add_argument("--sample", required=True, type=Path) + p_sub.add_argument("--buffer", required=True, type=Path) + p_sub.add_argument("--alpha", type=float, default=1.0) + p_sub.add_argument("--alpha-uncertainty", type=float, default=None) p_bl = sub.add_parser( "bl19b2-abs2d", @@ -374,6 +434,7 @@ def main() -> None: "x_col": result["x_col"], "i_col": result["i_col"], "err_col": result["err_col"], + "intensity_state": result.get("intensity_state"), }, ensure_ascii=False, ) @@ -382,25 +443,51 @@ def main() -> None: if args.command == "estimate-k": try: - q_meas, i_meas = _read_profile_for_estimate( - args.meas, - q_col=args.q_col, - i_col=args.i_col, - profile_label="measured", - ) - q_ref, i_ref = _read_profile_for_estimate( - args.ref, - q_col=args.ref_q_col, - i_col=args.ref_i_col, - profile_label="reference", + measured = _apply_declared_intensity_state( + _read_profile_for_estimate( + args.meas, + q_col=args.q_col, + i_col=args.i_col, + profile_label="measured", + ), + args.intensity_state, ) - out = estimate_k_factor_robust( - q_meas=q_meas, - i_meas_per_cm=i_meas, - q_ref=q_ref, - i_ref=i_ref, - q_window=(args.qmin, args.qmax), + assessment = require_relative_input_for_absolute_scaling( + measured, profile_name=str(args.meas) ) + i_meas = profile_intensity(measured) + if args.thickness_cm is not None: + thickness_cm = float(args.thickness_cm) + if not math.isfinite(thickness_cm) or thickness_cm <= 0: + raise ValueError("--thickness-cm must be finite and > 0") + if "thickness" in assessment.corrections_applied: + raise ValueError( + "measured profile already records thickness; " + "refuse a second --thickness-cm" + ) + i_meas = i_meas / thickness_cm + if args.ref is None: + if args.ref_q_col is not None or args.ref_i_col is not None: + raise ValueError("--ref-q-col and --ref-i-col require --ref") + out = estimate_k_factor_robust( + q_meas=measured["x"], + i_meas_per_cm=i_meas, + q_window=(args.qmin, args.qmax), + ) + else: + reference = _read_profile_for_estimate( + args.ref, + q_col=args.ref_q_col, + i_col=args.ref_i_col, + profile_label="reference", + ) + out = estimate_k_factor_robust( + q_meas=measured["x"], + i_meas_per_cm=i_meas, + q_ref=reference["x"], + i_ref=profile_intensity(reference), + q_window=(args.qmin, args.qmax), + ) except ValueError as exc: _die(f"estimate-k failed: {exc}") print( @@ -425,6 +512,38 @@ def main() -> None: ) return + if args.command == "subtract-buffer": + try: + sample = read_external_1d_profile(args.sample) + buffer_profile = read_external_1d_profile(args.buffer) + result = subtract_buffer( + sample["x"], + profile_intensity(sample), + profile_uncertainty(sample), + buffer_profile["x"], + profile_intensity(buffer_profile), + profile_uncertainty(buffer_profile), + alpha=args.alpha, + alpha_uncertainty=args.alpha_uncertainty, + sample_profile=sample, + buffer_profile=buffer_profile, + ) + except ValueError as exc: + _die(f"subtract-buffer failed: {exc}") + print( + json.dumps( + { + "points": int(result.q.size), + "alpha": result.alpha, + "alpha_uncertainty": result.alpha_uncertainty, + "high_q_residual_mean": result.high_q_residual_mean, + "high_q_check_passed": result.high_q_check_passed, + }, + ensure_ascii=False, + ) + ) + return + if args.command in {"bl19b2-abs2d", "bl19b2-abs2d-v1-legacy"}: from .workflows.bl19b2_abs2d import BL19B2Abs2DConfig, run_bl19b2_abs2d @@ -475,7 +594,8 @@ def main() -> None: calibration_background_monitor_relative_standard_uncertainty=( args.calibration_background_monitor_relative_standard_uncertainty ), - system_coverage_factor=args.system_coverage_factor, mu_relative_standard_uncertainty=args.mu_relative_standard_uncertainty, + system_coverage_factor=args.system_coverage_factor, + mu_relative_standard_uncertainty=args.mu_relative_standard_uncertainty, alpha_standard_uncertainty=args.alpha_standard_uncertainty, alpha=args.alpha, q_window=(args.qmin, args.qmax), diff --git a/src/saxsabs/core/buffer_subtraction.py b/src/saxsabs/core/buffer_subtraction.py index 916a339..4543340 100644 --- a/src/saxsabs/core/buffer_subtraction.py +++ b/src/saxsabs/core/buffer_subtraction.py @@ -14,10 +14,13 @@ from __future__ import annotations import logging +from collections.abc import Mapping from dataclasses import dataclass import numpy as np +from saxsabs.core.intensity_state import require_absolute_input_for_buffer_subtraction + logger = logging.getLogger(__name__) @@ -197,6 +200,8 @@ def subtract_buffer( high_q_diag: tuple[float, float] = (0.15, 0.25), *, alpha_uncertainty: float | None = None, + sample_profile: Mapping[str, object] | None = None, + buffer_profile: Mapping[str, object] | None = None, ) -> BufferSubtractionResult: """Subtract a buffer/solvent curve from a sample curve. @@ -217,11 +222,25 @@ def subtract_buffer( explicitly treated as exact. high_q_diag : tuple[float, float] *q* window for the high-*q* residual check. + sample_profile, buffer_profile + Intensity-state provenance for each curve. Both are required; unlabeled + arrays are refused. Returns ------- BufferSubtractionResult """ + if sample_profile is None or buffer_profile is None: + raise ValueError( + "subtract_buffer requires sample_profile and buffer_profile with " + "explicit absolute_cm^-1 intensity_state and a cm^-1 intensity_unit" + ) + require_absolute_input_for_buffer_subtraction( + sample_profile, profile_name="sample" + ) + require_absolute_input_for_buffer_subtraction( + buffer_profile, profile_name="buffer" + ) validate_alpha(alpha) try: q_lo, q_hi = (float(value) for value in high_q_diag) diff --git a/src/saxsabs/core/intensity_state.py b/src/saxsabs/core/intensity_state.py index 3d193d3..402f434 100644 --- a/src/saxsabs/core/intensity_state.py +++ b/src/saxsabs/core/intensity_state.py @@ -25,6 +25,15 @@ class IntensityState(str, Enum): ABSOLUTE_CORRECTIONS = frozenset({"thickness", "k"}) +CM_INV_UNIT_TOKENS = frozenset({"1cm", "cm1", "cminverse", "percm"}) + + +def is_cm_inv_intensity_unit(value: object) -> bool: + """Return True only for explicit cm^-1 unit strings, not a bare 'absolute'.""" + + return _normalized_token(value) in CM_INV_UNIT_TOKENS + + KNOWN_CORRECTIONS = frozenset( { "dark", @@ -138,7 +147,6 @@ def _state_from_metadata(value: object) -> IntensityState | None: "counts": IntensityState.RAW_COUNTS, "absolutecm1": IntensityState.ABSOLUTE_CM_INV, "absolute1cm": IntensityState.ABSOLUTE_CM_INV, - "absolute": IntensityState.ABSOLUTE_CM_INV, "ambiguous": IntensityState.AMBIGUOUS, "unknown": IntensityState.AMBIGUOUS, } @@ -207,7 +215,7 @@ def assess_intensity_state(profile: Mapping[str, object]) -> IntensityStateAsses unit = _normalized_token( profile.get("intensity_unit", provenance.get("intensity_unit", "")) ) - if unit in {"1cm", "cm1", "cminverse", "percm"}: + if is_cm_inv_intensity_unit(unit): semantic_states.add(IntensityState.ABSOLUTE_CM_INV) evidence.append(f"unit:{profile.get('intensity_unit', provenance.get('intensity_unit'))}") @@ -309,7 +317,7 @@ def require_absolute_input_for_buffer_subtraction( provenance = profile.get("operator_provenance") provenance = provenance if isinstance(provenance, Mapping) else {} raw_unit = profile.get("intensity_unit", provenance.get("intensity_unit", "")) - if _normalized_token(raw_unit) not in {"1cm", "cm1", "cminverse", "percm"}: + if not is_cm_inv_intensity_unit(raw_unit): raise ValueError(f"{profile_name}: buffer intensity_unit must be 1/cm") corrections = set(assessment.corrections_applied) missing = set(ABSOLUTE_CORRECTIONS) - corrections diff --git a/src/saxsabs/io/__init__.py b/src/saxsabs/io/__init__.py index 1721bef..9d845e0 100644 --- a/src/saxsabs/io/__init__.py +++ b/src/saxsabs/io/__init__.py @@ -2,6 +2,8 @@ extract_acquisition_timestamp, parse_header_values, parse_header_values_with_meta, + profile_intensity, + profile_uncertainty, read_cansas1d_xml, read_external_1d_profile, read_nxcansas_h5, @@ -20,6 +22,8 @@ "parse_header_values_with_meta", "extract_acquisition_timestamp", "read_external_1d_profile", + "profile_intensity", + "profile_uncertainty", "read_cansas1d_xml", "read_nxcansas_h5", "write_cansas1d_xml", diff --git a/src/saxsabs/io/parsers.py b/src/saxsabs/io/parsers.py index fa0014e..49cabbf 100644 --- a/src/saxsabs/io/parsers.py +++ b/src/saxsabs/io/parsers.py @@ -27,6 +27,8 @@ import numpy as np import pandas as pd +from saxsabs.core.intensity_state import IntensityState, assess_intensity_state + FLOAT_PATTERN = re.compile(r"[-+]?\d*\.?\d+(?:[eE][-+]?\d+)?") COMMA_THOUSANDS_PATTERN = re.compile(r"(? np.ndarray: + """Return the intensity array without assuming a relative or absolute label.""" + + for key in ("intensity", "i_abs", "i_rel"): + if key in profile: + return np.asarray(profile[key], dtype=np.float64) + raise KeyError("profile has no intensity array") + + +def profile_uncertainty(profile: dict[str, Any]) -> np.ndarray: + """Return the uncertainty array without assuming a relative or absolute label.""" + + for key in ("uncertainty", "err_abs", "err_rel"): + if key in profile: + return np.asarray(profile[key], dtype=np.float64) + raise KeyError("profile has no uncertainty array") + + +def _attach_intensity_arrays( + payload: dict[str, Any], + intensity: np.ndarray, + uncertainty: np.ndarray, +) -> dict[str, Any]: + """Expose intensity under a key that matches the assessed scientific state.""" + + payload["intensity"] = intensity + payload["uncertainty"] = uncertainty + assessment = assess_intensity_state(payload) + payload["intensity_state"] = assessment.state.value + if assessment.state is IntensityState.ABSOLUTE_CM_INV: + payload["i_abs"] = intensity + payload["err_abs"] = uncertainty + elif assessment.state is IntensityState.RELATIVE: + payload["i_rel"] = intensity + payload["err_rel"] = uncertainty + return payload + + def _operator_provenance_key(value: object) -> str | None: normalized = re.sub(r"[^a-z0-9]+", "", str(value).strip().lower()) return _OPERATOR_PROVENANCE_ALIASES.get(normalized) @@ -438,23 +478,23 @@ def read_external_1d_profile(path: str | Path) -> dict[str, Any]: ) x = pd.to_numeric(df[x_col], errors="coerce").to_numpy(dtype=np.float64, na_value=np.nan) - i_rel = pd.to_numeric(df[i_col], errors="coerce").to_numpy(dtype=np.float64, na_value=np.nan) - mask = np.isfinite(x) & np.isfinite(i_rel) + intensity = pd.to_numeric(df[i_col], errors="coerce").to_numpy(dtype=np.float64, na_value=np.nan) + mask = np.isfinite(x) & np.isfinite(intensity) if int(mask.sum()) < 3: continue x = x[mask] - i_rel = i_rel[mask] + intensity = intensity[mask] if err_named and err_col is not None: err = pd.to_numeric(df[err_col], errors="coerce").to_numpy(dtype=np.float64, na_value=np.nan)[mask] err = np.where(np.isfinite(err), err, np.nan) else: - err = np.full_like(i_rel, np.nan, dtype=np.float64) + err = np.full_like(intensity, np.nan, dtype=np.float64) order = np.argsort(x) x = x[order] - i_rel = i_rel[order] + intensity = intensity[order] err = err[order] pts = int(x.size) @@ -464,8 +504,8 @@ def read_external_1d_profile(path: str | Path) -> dict[str, Any]: best_rank = rank best = { "x": x, - "i_rel": i_rel, - "err_rel": err, + "intensity": intensity, + "uncertainty": err, "x_col": str(x_col), "i_col": str(i_col), "err_col": str(err_col) if err_named and err_col is not None else "", @@ -474,7 +514,9 @@ def read_external_1d_profile(path: str | Path) -> dict[str, Any]: if best is None: raise ValueError(f"Cannot identify valid numeric columns in {Path(path).name}") best["operator_provenance"] = _read_text_operator_provenance(p) - return best + intensity = np.asarray(best.pop("intensity"), dtype=np.float64) + uncertainty = np.asarray(best.pop("uncertainty"), dtype=np.float64) + return _attach_intensity_arrays(best, intensity, uncertainty) # --------------------------------------------------------------------------- @@ -486,8 +528,9 @@ def read_external_1d_profile(path: str | Path) -> dict[str, Any]: def read_cansas1d_xml(path: str | Path) -> dict[str, Any]: """Read a canSAS 1D XML file and return a profile dict. - Returns the same ``{"x", "i_rel", "err_rel", ...}`` dict as - :func:`read_external_1d_profile`. + Returns the same profile dict as :func:`read_external_1d_profile`, with + ``intensity`` always present and ``i_abs`` or ``i_rel`` only when the + assessed state matches. """ p = Path(path) tree = ET.parse(str(p)) @@ -529,7 +572,7 @@ def read_cansas1d_xml(path: str | Path) -> dict[str, Any]: raise ValueError(f"canSAS XML contains too few data points: {p.name}") x = np.asarray(q_vals, dtype=np.float64) - i_rel = np.asarray(i_vals, dtype=np.float64) + intensity = np.asarray(i_vals, dtype=np.float64) err = np.asarray(e_vals, dtype=np.float64) if e_vals else np.full_like(x, np.nan) operator_provenance: dict[str, str] = {} @@ -539,16 +582,18 @@ def read_cansas1d_xml(path: str | Path) -> dict[str, Any]: if key is not None and term.text is not None: operator_provenance[key] = term.text.strip() order = np.argsort(x) - return { - "x": x[order], - "i_rel": i_rel[order], - "err_rel": err[order], - "x_col": "Q", - "i_col": "I", - "err_col": "Idev", - "intensity_unit": intensity_unit, - "operator_provenance": operator_provenance, - } + return _attach_intensity_arrays( + { + "x": x[order], + "x_col": "Q", + "i_col": "I", + "err_col": "Idev", + "intensity_unit": intensity_unit, + "operator_provenance": operator_provenance, + }, + intensity[order], + err[order], + ) # --------------------------------------------------------------------------- @@ -558,7 +603,7 @@ def read_cansas1d_xml(path: str | Path) -> dict[str, Any]: def read_nxcansas_h5(path: str | Path) -> dict[str, Any]: """Read an NXcanSAS HDF5 file and return a profile dict. - Requires the ``h5py`` package. Returns the same dict format as + Requires the ``h5py`` package. Returns the same profile dict as :func:`read_external_1d_profile`. """ try: @@ -630,17 +675,17 @@ def _collect_operator_provenance(_name: str, item: Any) -> None: raise ValueError(f"Cannot find SASdata/Q,I datasets in {p.name}") x = np.asarray(q_ds, dtype=np.float64).ravel() - i_rel = np.asarray(i_ds, dtype=np.float64).ravel() + intensity = np.asarray(i_ds, dtype=np.float64).ravel() err = ( np.asarray(e_ds, dtype=np.float64).ravel() if e_ds is not None else np.full_like(x, np.nan) ) - if x.shape != i_rel.shape: + if x.shape != intensity.shape: raise ValueError( f"NXcanSAS dataset length mismatch in {p.name}: " - f"Q has {x.size} points, I has {i_rel.size}" + f"Q has {x.size} points, I has {intensity.size}" ) if err.shape != x.shape: raise ValueError( @@ -649,16 +694,18 @@ def _collect_operator_provenance(_name: str, item: Any) -> None: ) order = np.argsort(x) - return { - "x": x[order], - "i_rel": i_rel[order], - "err_rel": err[order], - "x_col": "Q", - "i_col": "I", - "err_col": "Idev" if e_ds is not None else "", - "intensity_unit": intensity_unit, - "operator_provenance": operator_provenance, - } + return _attach_intensity_arrays( + { + "x": x[order], + "x_col": "Q", + "i_col": "I", + "err_col": "Idev" if e_ds is not None else "", + "intensity_unit": intensity_unit, + "operator_provenance": operator_provenance, + }, + intensity[order], + err[order], + ) # --------------------------------------------------------------------------- diff --git a/src/saxsabs/io/writers.py b/src/saxsabs/io/writers.py index e7978e4..3e104e2 100644 --- a/src/saxsabs/io/writers.py +++ b/src/saxsabs/io/writers.py @@ -14,6 +14,12 @@ import numpy as np +from saxsabs.core.intensity_state import ( + IntensityState, + assess_intensity_state, + is_cm_inv_intensity_unit, +) + # --------------------------------------------------------------------------- # canSAS 1D XML (cansas1d:1.1) @@ -69,6 +75,27 @@ def _operator_provenance_from_metadata(metadata: dict[str, Any]) -> dict[str, st provenance[key] = str(value).strip() return provenance + +def _require_absolute_cm_inv_unit(metadata: dict[str, Any]) -> str: + """Stamp 1/cm only when the caller declared absolute cm^-1 intensity.""" + + assessment = assess_intensity_state(metadata) + if assessment.state is not IntensityState.ABSOLUTE_CM_INV: + raise ValueError( + "absolute-intensity writers require intensity_state=absolute_cm^-1 " + f"and a cm^-1 intensity_unit; assessed state is {assessment.state.value}" + ) + nested = metadata.get("operator_provenance") + nested = nested if isinstance(nested, dict) else {} + unit = metadata.get("intensity_unit", nested.get("intensity_unit", "")) + if not is_cm_inv_intensity_unit(unit): + raise ValueError( + "absolute-intensity writers refuse to stamp 1/cm without an explicit " + "cm^-1 intensity_unit" + ) + return "1/cm" + + def _prepare_profile_arrays( q: np.ndarray, i_abs: np.ndarray, @@ -122,6 +149,7 @@ def write_cansas1d_xml( The written file path. """ meta = metadata or {} + intensity_unit = _require_absolute_cm_inv_unit(meta) out = Path(path) root = ET.Element("SASroot") @@ -142,10 +170,10 @@ def write_cansas1d_xml( idata = ET.SubElement(sasdata, "Idata") qel = ET.SubElement(idata, "Q", unit="1/A") qel.text = f"{q_arr[idx]:.8g}" - iel = ET.SubElement(idata, "I", unit="1/cm") + iel = ET.SubElement(idata, "I", unit=intensity_unit) iel.text = f"{i_arr[idx]:.8g}" if e_arr is not None and np.isfinite(e_arr[idx]): - eel = ET.SubElement(idata, "Idev", unit="1/cm") + eel = ET.SubElement(idata, "Idev", unit=intensity_unit) eel.text = f"{e_arr[idx]:.8g}" # --- SASsample --- @@ -222,6 +250,7 @@ def write_nxcansas_h5( ) from exc meta = metadata or {} + intensity_unit = _require_absolute_cm_inv_unit(meta) out = Path(path) out.parent.mkdir(parents=True, exist_ok=True) q_arr, i_arr, e_arr = _prepare_profile_arrays(q, i_abs, err) @@ -247,11 +276,11 @@ def write_nxcansas_h5( ds_q.attrs["units"] = "1/angstrom" ds_i = data.create_dataset("I", data=i_arr) - ds_i.attrs["units"] = "1/cm" + ds_i.attrs["units"] = intensity_unit if e_arr is not None: ds_e = data.create_dataset("Idev", data=e_arr) - ds_e.attrs["units"] = "1/cm" + ds_e.attrs["units"] = intensity_unit # SASinstrument (minimal) inst = entry.create_group("sasinstrument01") diff --git a/tests/test_buffer_subtraction.py b/tests/test_buffer_subtraction.py index 83d2505..ed71953 100644 --- a/tests/test_buffer_subtraction.py +++ b/tests/test_buffer_subtraction.py @@ -9,6 +9,22 @@ validate_alpha, ) +ABS_PROFILE = { + "intensity_state": "absolute_cm^-1", + "intensity_unit": "1/cm", + "i_col": "I_abs_cm^-1", + "operator_provenance": { + "intensity_state": "absolute_cm^-1", + "corrections_applied": '["k","thickness"]', + }, +} + + +def _sub(*args, **kwargs): + kwargs.setdefault("sample_profile", ABS_PROFILE) + kwargs.setdefault("buffer_profile", ABS_PROFILE) + return subtract_buffer(*args, **kwargs) + class TestSubtractBuffer: def _make_data(self, n=100): @@ -21,7 +37,7 @@ def _make_data(self, n=100): def test_basic_subtraction(self): q, i_s, e_s, i_b, e_b = self._make_data() - result = subtract_buffer(q, i_s, e_s, q, i_b, e_b, alpha=1.0) + result = _sub(q, i_s, e_s, q, i_b, e_b, alpha=1.0) assert isinstance(result, BufferSubtractionResult) expected = i_s - i_b np.testing.assert_allclose(result.i_subtracted, expected, rtol=1e-10) @@ -29,14 +45,14 @@ def test_basic_subtraction(self): def test_alpha_scaling(self): q, i_s, e_s, i_b, e_b = self._make_data() alpha = 0.9 - result = subtract_buffer(q, i_s, e_s, q, i_b, e_b, alpha=alpha) + result = _sub(q, i_s, e_s, q, i_b, e_b, alpha=alpha) expected = i_s - alpha * i_b np.testing.assert_allclose(result.i_subtracted, expected, rtol=1e-10) def test_error_propagation(self): q, i_s, e_s, i_b, e_b = self._make_data() alpha = 1.0 - result = subtract_buffer( + result = _sub( q, i_s, e_s, @@ -53,7 +69,7 @@ def test_error_propagation(self): def test_missing_alpha_uncertainty_keeps_combined_uncertainty_unknown(self): q, i_s, e_s, i_b, e_b = self._make_data(n=5) - result = subtract_buffer(q, i_s, e_s, q, i_b, e_b) + result = _sub(q, i_s, e_s, q, i_b, e_b) assert result.alpha_uncertainty is None expected_statistical = np.sqrt(e_s**2 + e_b**2) @@ -68,7 +84,7 @@ def test_missing_input_uncertainty_remains_unknown(self, missing): else: e_b = None - result = subtract_buffer( + result = _sub( q, i_s, e_s, q, i_b, e_b, alpha_uncertainty=0.0 ) @@ -78,7 +94,7 @@ def test_partial_unknown_input_uncertainty_is_not_replaced_by_zero(self): q, i_s, e_s, i_b, e_b = self._make_data(n=5) e_s[2] = np.nan - result = subtract_buffer( + result = _sub( q, i_s, e_s, q, i_b, e_b, alpha_uncertainty=0.0 ) @@ -88,7 +104,7 @@ def test_partial_unknown_input_uncertainty_is_not_replaced_by_zero(self): def test_interpolated_error_propagates_variance_with_squared_weights(self): q_s = np.array([1.0]) q_b = np.array([0.0, 2.0]) - result = subtract_buffer( + result = _sub( q_s, np.array([10.0]), np.array([0.4]), @@ -103,7 +119,7 @@ def test_interpolated_error_propagates_variance_with_squared_weights(self): def test_alpha_uncertainty_is_propagated_from_buffer_intensity(self): q = np.array([0.01, 0.02, 0.03]) - result = subtract_buffer( + result = _sub( q, np.full(3, 10.0), np.full(3, 0.1), @@ -123,7 +139,7 @@ def test_alpha_uncertainty_is_propagated_from_buffer_intensity(self): def test_invalid_alpha_uncertainty_raises(self, alpha_uncertainty): q, i_s, e_s, i_b, e_b = self._make_data(n=5) with pytest.raises(ValueError, match="alpha_uncertainty"): - subtract_buffer( + _sub( q, i_s, e_s, @@ -141,7 +157,7 @@ def test_interpolation_different_grids(self): i_b = np.ones(200) * 3.0 err_s = np.full(n, 0.1) err_b = np.full(200, 0.05) - result = subtract_buffer(q_s, i_s, err_s, q_b, i_b, err_b, alpha=1.0) + result = _sub(q_s, i_s, err_s, q_b, i_b, err_b, alpha=1.0) assert result.q.shape == q_s.shape np.testing.assert_allclose(result.i_subtracted, 7.0, atol=0.1) @@ -150,7 +166,7 @@ def test_close_but_distinct_q_grids_are_still_interpolated(self): q_b = np.array([0.099999, 0.200001, 0.300001]) i_b = 1.0e9 * q_b - result = subtract_buffer( + result = _sub( q_s, np.zeros(3), np.zeros(3), @@ -171,7 +187,7 @@ def test_interpolation_outside_buffer_range_raises(self): err_b = np.full(3, 0.05) try: - subtract_buffer(q_s, i_s, err_s, q_b, i_b, err_b, alpha=1.0) + _sub(q_s, i_s, err_s, q_b, i_b, err_b, alpha=1.0) except ValueError as exc: assert "outside buffer q range" in str(exc) else: @@ -192,7 +208,7 @@ def test_negative_finite_errors_raise_before_non_finite_replacement( i_b = np.ones(3) * 2.0 with pytest.raises(ValueError, match=message): - subtract_buffer(q, i_s, err_sample, q, i_b, err_buffer, alpha=1.0) + _sub(q, i_s, err_sample, q, i_b, err_buffer, alpha=1.0) class TestValidateAlpha: @@ -218,7 +234,7 @@ def test_alpha_negative_raises(self): def test_subtract_buffer_preserves_legacy_positional_high_q_window(): q = np.array([0.10, 0.20, 0.30], dtype=float) - result = subtract_buffer( + result = _sub( q, np.array([3.0, 3.0, 3.0]), np.zeros(3), @@ -246,4 +262,10 @@ def test_subtract_buffer_rejects_infinite_uncertainty(field): kwargs[field][1] = np.inf with pytest.raises(ValueError, match=field): - subtract_buffer(**kwargs) + _sub(**kwargs) + + +def test_subtract_buffer_refuses_unlabeled_profiles(): + q = np.array([0.01, 0.02, 0.03]) + with pytest.raises(ValueError, match="sample_profile and buffer_profile"): + subtract_buffer(q, np.ones(3), np.ones(3), q, np.ones(3), np.ones(3)) diff --git a/tests/test_cli.py b/tests/test_cli.py index 01f3fe5..689e8f7 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -65,6 +65,8 @@ def test_cli_estimate_k(tmp_path: Path, capsys: pytest.CaptureFixture[str], monk str(meas), "--ref", str(ref), + "--intensity-state", + "relative", "--qmin", "0.01", "--qmax", @@ -100,7 +102,16 @@ def test_cli_estimate_k_accepts_common_intensity_column_names( monkeypatch.setattr( sys, "argv", - ["saxsabs", "estimate-k", "--meas", str(meas), "--ref", str(ref)], + [ + "saxsabs", + "estimate-k", + "--meas", + str(meas), + "--ref", + str(ref), + "--intensity-state", + "relative", + ], ) main() out = json.loads(capsys.readouterr().out) @@ -115,18 +126,27 @@ def test_cli_estimate_k_accepts_scattering_column_names_with_units( meas = tmp_path / "meas.csv" ref = tmp_path / "ref.csv" meas.write_text( - "Q_A^-1,I_abs\n0.01,17.1\n0.02,15.4\n0.05,13.4\n0.10,11.8\n", + "Q_A^-1,I_rel\n0.01,17.1\n0.02,15.4\n0.05,13.4\n0.10,11.8\n", encoding="utf-8", ) ref.write_text( - "Q_A^-1,I_abs\n0.01,34.2\n0.02,30.8\n0.05,26.8\n0.10,23.6\n", + "Q_A^-1,I_rel\n0.01,34.2\n0.02,30.8\n0.05,26.8\n0.10,23.6\n", encoding="utf-8", ) monkeypatch.setattr( sys, "argv", - ["saxsabs", "estimate-k", "--meas", str(meas), "--ref", str(ref)], + [ + "saxsabs", + "estimate-k", + "--meas", + str(meas), + "--ref", + str(ref), + "--intensity-state", + "relative", + ], ) main() out = json.loads(capsys.readouterr().out) @@ -167,6 +187,8 @@ def test_cli_estimate_k_accepts_explicit_column_overrides( "q_ref", "--ref-i-col", "absolute", + "--intensity-state", + "relative", ], ) main() @@ -201,6 +223,8 @@ def test_cli_estimate_k_invalid_override_lists_available_columns( "missing_q", "--i-col", "intensity", + "--intensity-state", + "relative", ], ) @@ -224,6 +248,134 @@ def test_cli_parse_external1d(tmp_path: Path, capsys: pytest.CaptureFixture[str] assert out["x_col"] == "q" assert out["i_col"] == "i" assert out["err_col"] == "err" + assert out["intensity_state"] == "ambiguous" + + +def test_cli_estimate_k_refuses_unlabeled_intensity( + tmp_path: Path, + capsys: pytest.CaptureFixture[str], + monkeypatch: pytest.MonkeyPatch, +): + meas = tmp_path / "meas.csv" + ref = tmp_path / "ref.csv" + meas.write_text("q,i\n0.01,17.1\n0.02,15.4\n0.05,13.4\n0.10,11.8\n", encoding="utf-8") + ref.write_text("q,i\n0.01,34.2\n0.02,30.8\n0.05,26.8\n0.10,23.6\n", encoding="utf-8") + monkeypatch.setattr( + sys, + "argv", + ["saxsabs", "estimate-k", "--meas", str(meas), "--ref", str(ref)], + ) + + with pytest.raises(SystemExit) as exc_info: + main() + + assert exc_info.value.code == 1 + assert "ambiguous" in capsys.readouterr().err.lower() + + +def test_cli_estimate_k_thickness_cm_converts_to_per_cm( + tmp_path: Path, + capsys: pytest.CaptureFixture[str], + monkeypatch: pytest.MonkeyPatch, +): + meas = tmp_path / "meas.csv" + ref = tmp_path / "ref.csv" + meas.write_text( + "q,i\n0.01,1.71\n0.02,1.54\n0.05,1.34\n0.10,1.18\n", + encoding="utf-8", + ) + ref.write_text("q,i\n0.01,34.2\n0.02,30.8\n0.05,26.8\n0.10,23.6\n", encoding="utf-8") + monkeypatch.setattr( + sys, + "argv", + [ + "saxsabs", + "estimate-k", + "--meas", + str(meas), + "--ref", + str(ref), + "--intensity-state", + "relative", + "--thickness-cm", + "0.1", + ], + ) + main() + out = json.loads(capsys.readouterr().out) + assert out["k_factor"] == pytest.approx(2.0, rel=1e-6) + + +def _write_absolute_profile(path: Path, intensities: str) -> None: + path.write_text( + "# intensity_state: absolute_cm^-1\n" + "# intensity_unit: 1/cm\n" + '# corrections_applied: ["k","thickness"]\n' + "q,i\n" + f"{intensities}", + encoding="utf-8", + ) + + +def test_cli_subtract_buffer( + tmp_path: Path, + capsys: pytest.CaptureFixture[str], + monkeypatch: pytest.MonkeyPatch, +): + sample = tmp_path / "sample.csv" + buffer = tmp_path / "buffer.csv" + _write_absolute_profile(sample, "0.01,12.0\n0.02,11.0\n0.20,6.0\n") + _write_absolute_profile(buffer, "0.01,2.0\n0.02,2.0\n0.20,2.0\n") + monkeypatch.setattr( + sys, + "argv", + [ + "saxsabs", + "subtract-buffer", + "--sample", + str(sample), + "--buffer", + str(buffer), + "--alpha", + "1.0", + ], + ) + main() + out = json.loads(capsys.readouterr().out) + assert out["points"] == 3 + assert out["alpha"] == pytest.approx(1.0) + + +def test_cli_subtract_buffer_refuses_unlabeled_profiles( + tmp_path: Path, + capsys: pytest.CaptureFixture[str], + monkeypatch: pytest.MonkeyPatch, +): + sample = tmp_path / "sample.csv" + buffer = tmp_path / "buffer.csv" + sample.write_text("q,i\n0.01,12.0\n0.02,11.0\n0.20,6.0\n", encoding="utf-8") + buffer.write_text("q,i\n0.01,2.0\n0.02,2.0\n0.20,2.0\n", encoding="utf-8") + monkeypatch.setattr( + sys, + "argv", + ["saxsabs", "subtract-buffer", "--sample", str(sample), "--buffer", str(buffer)], + ) + + with pytest.raises(SystemExit) as exc_info: + main() + + assert exc_info.value.code == 1 + assert "absolute" in capsys.readouterr().err.lower() + + +def test_cli_version(capsys: pytest.CaptureFixture[str], monkeypatch: pytest.MonkeyPatch): + from saxsabs import __version__ + + monkeypatch.setattr(sys, "argv", ["saxsabs", "--version"]) + with pytest.raises(SystemExit) as exc_info: + main() + assert exc_info.value.code == 0 + assert f"saxsabs {__version__}" in capsys.readouterr().out def test_cli_bl19b2_abs2d_passes_pydidas_yaml_and_mask_to_workflow( diff --git a/tests/test_intensity_state.py b/tests/test_intensity_state.py index 4b51934..d9df1ae 100644 --- a/tests/test_intensity_state.py +++ b/tests/test_intensity_state.py @@ -22,6 +22,15 @@ def test_absolute_column_is_rejected_before_k_or_thickness_is_reapplied(): require_relative_input_for_absolute_scaling(profile, profile_name="sample.dat") +def test_unitless_absolute_metadata_is_ambiguous(): + assessment = assess_intensity_state( + {"i_col": "I", "operator_provenance": {"intensity_state": "absolute"}} + ) + + assert assessment.state is IntensityState.AMBIGUOUS + assert "invalid_metadata:intensity_state" in assessment.evidence + + def test_absolute_unit_is_detected_even_with_generic_i_column(): assessment = assess_intensity_state({"i_col": "I", "intensity_unit": "1/cm"}) @@ -40,6 +49,7 @@ def test_invalid_explicit_state_is_ambiguous_even_with_absolute_unit_evidence(): assert assessment.state is IntensityState.AMBIGUOUS assert "invalid_metadata:intensity_state" in assessment.evidence + def test_relative_provenance_allows_absolute_scaling(): profile = { "i_col": "I", diff --git a/tests/test_io_formats.py b/tests/test_io_formats.py index 0714437..17f4e46 100644 --- a/tests/test_io_formats.py +++ b/tests/test_io_formats.py @@ -9,6 +9,16 @@ from saxsabs.io.parsers import read_cansas1d_xml, read_external_1d_profile +ABS_META = { + "intensity_state": "absolute_cm^-1", + "intensity_unit": "1/cm", +} + + +def _meta(**extra): + return {**ABS_META, **extra} + + # --------------------------------------------------------------------------- # canSAS 1D XML round-trip # --------------------------------------------------------------------------- @@ -22,20 +32,21 @@ def _make_data(self, n=50): def test_write_read_roundtrip(self, tmp_path): q, i_abs, err = self._make_data() xml_path = tmp_path / "test.xml" - write_cansas1d_xml(xml_path, q, i_abs, err, metadata={"title": "round-trip test"}) + write_cansas1d_xml(xml_path, q, i_abs, err, metadata=_meta(title="round-trip test")) assert xml_path.exists() result = read_cansas1d_xml(xml_path) - assert "x" in result - assert "i_rel" in result + assert "i_abs" in result + assert "i_rel" not in result + assert result["intensity_state"] == "absolute_cm^-1" np.testing.assert_allclose(result["x"], q, rtol=1e-6) - np.testing.assert_allclose(result["i_rel"], i_abs, rtol=1e-6) - np.testing.assert_allclose(result["err_rel"], err, rtol=1e-6) + np.testing.assert_allclose(result["i_abs"], i_abs, rtol=1e-6) + np.testing.assert_allclose(result["err_abs"], err, rtol=1e-6) def test_write_no_error(self, tmp_path): q, i_abs, _ = self._make_data() xml_path = tmp_path / "no_err.xml" - write_cansas1d_xml(xml_path, q, i_abs) + write_cansas1d_xml(xml_path, q, i_abs, metadata=ABS_META) result = read_cansas1d_xml(xml_path) np.testing.assert_allclose(result["x"], q, rtol=1e-6) @@ -43,9 +54,11 @@ def test_auto_detect_xml_extension(self, tmp_path): """read_external_1d_profile should auto-detect .xml files.""" q, i_abs, err = self._make_data() xml_path = tmp_path / "auto.xml" - write_cansas1d_xml(xml_path, q, i_abs, err) + write_cansas1d_xml(xml_path, q, i_abs, err, metadata=ABS_META) result = read_external_1d_profile(str(xml_path)) np.testing.assert_allclose(result["x"], q, rtol=1e-6) + assert "i_abs" in result + assert "i_rel" not in result def test_metadata_preserved(self, tmp_path): q, i_abs, err = self._make_data(10) @@ -57,14 +70,14 @@ def test_metadata_preserved(self, tmp_path): "sdd_m": 2.0, "sample_name": "glass", } - out = write_cansas1d_xml(xml_path, q, i_abs, err, metadata=meta) + out = write_cansas1d_xml(xml_path, q, i_abs, err, metadata=_meta(**meta)) assert out == xml_path def test_write_includes_schema_required_notes(self, tmp_path): q, i_abs, err = self._make_data(3) xml_path = tmp_path / "schema-required-notes.xml" - write_cansas1d_xml(xml_path, q, i_abs, err) + write_cansas1d_xml(xml_path, q, i_abs, err, metadata=ABS_META) namespace = {"cansas": "urn:cansas1d:1.1"} root = ET.parse(xml_path).getroot() @@ -83,25 +96,45 @@ def test_inherited_thickness_provenance_roundtrip(self, tmp_path): q, i_abs, err, - metadata={ - "thickness_cm": "0.1", - "thickness_source": "upstream sample cell record", - }, + metadata=_meta( + thickness_cm="0.1", + thickness_source="upstream sample cell record", + ), ) provenance = read_cansas1d_xml(xml_path)["operator_provenance"] assert provenance["thickness_cm"] == "0.1" assert provenance["thickness_source"] == "upstream sample cell record" + def test_write_refuses_unlabeled_intensity(self, tmp_path): + q, i_abs, err = self._make_data(3) + with pytest.raises(ValueError, match="intensity_state=absolute_cm\\^-1"): + write_cansas1d_xml(tmp_path / "unlabeled.xml", q, i_abs, err) + + def test_write_refuses_unitless_absolute_label(self, tmp_path): + q, i_abs, err = self._make_data(3) + with pytest.raises(ValueError, match="absolute_cm\\^-1|cm\\^-1 intensity_unit"): + write_cansas1d_xml( + tmp_path / "unitless.xml", + q, + i_abs, + err, + metadata={"intensity_state": "absolute"}, + ) + def test_write_shape_mismatch_raises(self, tmp_path): xml_path = tmp_path / "bad.xml" try: - write_cansas1d_xml(xml_path, np.array([0.1, 0.2, 0.3]), np.array([10.0, 9.0])) + write_cansas1d_xml( + xml_path, + np.array([0.1, 0.2, 0.3]), + np.array([10.0, 9.0]), + metadata=ABS_META, + ) except ValueError as exc: assert "same shape" in str(exc) else: raise AssertionError("Expected ValueError for mismatched q/i shapes") - @pytest.mark.parametrize("bad_value", [np.nan, np.inf, -np.inf]) def test_write_rejects_nonfinite_q(self, tmp_path, bad_value): with pytest.raises(ValueError, match="q must contain only finite values"): @@ -109,6 +142,7 @@ def test_write_rejects_nonfinite_q(self, tmp_path, bad_value): tmp_path / "bad-q.xml", np.array([0.1, bad_value]), np.array([10.0, 9.0]), + metadata=ABS_META, ) @pytest.mark.parametrize("bad_value", [np.nan, np.inf, -np.inf]) @@ -118,10 +152,10 @@ def test_write_rejects_nonfinite_intensity(self, tmp_path, bad_value): tmp_path / "bad-i.xml", np.array([0.1, 0.2]), np.array([10.0, bad_value]), + metadata=ABS_META, ) - # --------------------------------------------------------------------------- # NXcanSAS HDF5 round-trip (skip if h5py unavailable) # --------------------------------------------------------------------------- @@ -139,21 +173,24 @@ def _make_data(self, n=50): def test_write_read_roundtrip(self, tmp_path): q, i_abs, err = self._make_data() h5_path = tmp_path / "test.h5" - write_nxcansas_h5(h5_path, q, i_abs, err, metadata={"title": "h5 round-trip"}) + write_nxcansas_h5(h5_path, q, i_abs, err, metadata=_meta(title="h5 round-trip")) assert h5_path.exists() result = read_nxcansas_h5(h5_path) + assert "i_abs" in result + assert "i_rel" not in result np.testing.assert_allclose(result["x"], q, rtol=1e-10) - np.testing.assert_allclose(result["i_rel"], i_abs, rtol=1e-10) - np.testing.assert_allclose(result["err_rel"], err, rtol=1e-10) + np.testing.assert_allclose(result["i_abs"], i_abs, rtol=1e-10) + np.testing.assert_allclose(result["err_abs"], err, rtol=1e-10) def test_auto_detect_h5_extension(self, tmp_path): """read_external_1d_profile should auto-detect .h5 files.""" q, i_abs, err = self._make_data() h5_path = tmp_path / "auto.h5" - write_nxcansas_h5(h5_path, q, i_abs, err) + write_nxcansas_h5(h5_path, q, i_abs, err, metadata=ABS_META) result = read_external_1d_profile(str(h5_path)) np.testing.assert_allclose(result["x"], q, rtol=1e-10) + assert "i_abs" in result def test_inherited_thickness_provenance_roundtrip(self, tmp_path): q, i_abs, err = self._make_data(10) @@ -163,10 +200,10 @@ def test_inherited_thickness_provenance_roundtrip(self, tmp_path): q, i_abs, err, - metadata={ - "thickness_cm": "0.1", - "thickness_source": "upstream sample cell record", - }, + metadata=_meta( + thickness_cm="0.1", + thickness_source="upstream sample cell record", + ), ) provenance = read_nxcansas_h5(h5_path)["operator_provenance"] assert provenance["thickness_cm"] == "0.1" @@ -179,6 +216,7 @@ def test_write_rejects_nonfinite_q(self, tmp_path, bad_value): tmp_path / "bad-q.h5", np.array([0.1, bad_value]), np.array([10.0, 9.0]), + metadata=ABS_META, ) @pytest.mark.parametrize("bad_value", [np.nan, np.inf, -np.inf]) @@ -188,26 +226,36 @@ def test_write_rejects_nonfinite_intensity(self, tmp_path, bad_value): tmp_path / "bad-i.h5", np.array([0.1, 0.2]), np.array([10.0, bad_value]), + metadata=ABS_META, ) - def test_no_error_dataset(self, tmp_path): q, i_abs, _ = self._make_data() h5_path = tmp_path / "no_err.h5" - write_nxcansas_h5(h5_path, q, i_abs) + write_nxcansas_h5(h5_path, q, i_abs, metadata=ABS_META) result = read_nxcansas_h5(h5_path) np.testing.assert_allclose(result["x"], q, rtol=1e-10) - assert np.all(np.isnan(result["err_rel"])) + assert np.all(np.isnan(result["err_abs"])) def test_write_shape_mismatch_raises(self, tmp_path): h5_path = tmp_path / "bad.h5" try: - write_nxcansas_h5(h5_path, np.array([0.1, 0.2, 0.3]), np.array([10.0, 9.0])) + write_nxcansas_h5( + h5_path, + np.array([0.1, 0.2, 0.3]), + np.array([10.0, 9.0]), + metadata=ABS_META, + ) except ValueError as exc: assert "same shape" in str(exc) else: raise AssertionError("Expected ValueError for mismatched q/i shapes") + def test_write_refuses_unlabeled_intensity(self, tmp_path): + q, i_abs, err = self._make_data(3) + with pytest.raises(ValueError, match="intensity_state=absolute_cm\\^-1"): + write_nxcansas_h5(tmp_path / "unlabeled.h5", q, i_abs, err) + def test_reader_rejects_malformed_dataset_lengths(self, tmp_path): h5_path = tmp_path / "malformed.h5" with h5py.File(h5_path, "w") as f: diff --git a/tests/test_parsers.py b/tests/test_parsers.py index 3dfbbd4..54fc566 100644 --- a/tests/test_parsers.py +++ b/tests/test_parsers.py @@ -77,7 +77,8 @@ def test_read_external_1d_profile_csv(tmp_path: Path): assert out["x"].size == 3 assert out["x_col"].lower() == "q" assert out["i_col"].lower() == "intensity" - assert np.isclose(out["i_rel"][0], 100.0) + assert np.isclose(out["intensity"][0], 100.0) + assert "i_rel" not in out def test_read_external_1d_profile_space_delimited(tmp_path: Path): @@ -92,7 +93,7 @@ def test_read_external_1d_profile_space_delimited(tmp_path: Path): out = read_external_1d_profile(f) assert out["x"].size == 3 - assert np.isfinite(out["err_rel"]).all() + assert np.isfinite(out["uncertainty"]).all() assert out["err_col"].lower() == "sigma" @@ -142,7 +143,8 @@ def test_read_external_1d_profile_prefers_intensity_over_id_column(tmp_path: Pat out = read_external_1d_profile(f) assert out["x_col"].lower() == "q" assert out["i_col"].lower() == "intensity" - np.testing.assert_allclose(out["i_rel"], [100.0, 90.0, 80.0]) + np.testing.assert_allclose(out["intensity"], [100.0, 90.0, 80.0]) + assert "i_rel" not in out def test_read_external_1d_profile_unnamed_third_column_not_treated_as_error(tmp_path: Path): @@ -157,7 +159,7 @@ def test_read_external_1d_profile_unnamed_third_column_not_treated_as_error(tmp_ out = read_external_1d_profile(f) assert out["x"].size == 3 assert out["err_col"] == "" - assert np.all(np.isnan(out["err_rel"])) + assert np.all(np.isnan(out["uncertainty"])) def test_read_external_1d_profile_prefers_combined_over_earlier_statistical_column( @@ -175,4 +177,4 @@ def test_read_external_1d_profile_prefers_combined_over_earlier_statistical_colu out = read_external_1d_profile(f) assert out["err_col"] == "Error_CombinedStandard_cm^-1" - np.testing.assert_allclose(out["err_rel"], [1.5, 2.5, 3.5]) + np.testing.assert_allclose(out["uncertainty"], [1.5, 2.5, 3.5]) diff --git a/tests/test_workbench_scientific.py b/tests/test_workbench_scientific.py index af58397..539b583 100644 --- a/tests/test_workbench_scientific.py +++ b/tests/test_workbench_scientific.py @@ -242,7 +242,7 @@ def test_tab3_profile_parser_prefers_combined_over_earlier_statistical_column( profile = app.read_external_1d_profile(profile_path) assert profile["err_col"] == "Error_CombinedStandard_cm^-1" - np.testing.assert_allclose(profile["err_rel"], [1.5, 2.5, 3.5]) + np.testing.assert_allclose(profile["uncertainty"], [1.5, 2.5, 3.5]) def test_tab3_formal_external_profile_requires_matching_operator_provenance(tmp_path): @@ -1288,13 +1288,23 @@ def test_tab3_workbench_parser_roundtrips_cansas_xml(tmp_path): q = np.array([0.01, 0.02, 0.03]) intensity = np.array([10.0, 9.0, 8.0]) error = np.array([0.1, 0.2, 0.3]) - module.write_cansas1d_xml(output, q, intensity, error) + module.write_cansas1d_xml( + output, + q, + intensity, + error, + metadata={ + "intensity_state": "absolute_cm^-1", + "intensity_unit": "1/cm", + }, + ) profile = app.read_external_1d_profile(output) np.testing.assert_allclose(profile["x"], q) - np.testing.assert_allclose(profile["i_rel"], intensity) - np.testing.assert_allclose(profile["err_rel"], error) + np.testing.assert_allclose(profile["i_abs"], intensity) + np.testing.assert_allclose(profile["err_abs"], error) + assert "i_rel" not in profile assert profile["x_col"] == "Q" assert profile["i_col"] == "I" assert profile["err_col"] == "Idev" @@ -1607,7 +1617,7 @@ def counted_read(path): sample_output = tmp_path / "output" / "processed_external_1d_abs" / "sample_a.dat" output_profile = app.read_external_1d_profile(sample_output) - assert output_profile["i_rel"][0] == pytest.approx(24.5) + assert output_profile["i_abs"][0] == pytest.approx(24.5) assert output_profile["operator_provenance"]["buffer_alpha"] == "0.5" assert output_profile["operator_provenance"]["buffer_alpha_uncertainty"] == "0.05" assert output_profile["operator_provenance"]["buffer_source_name"] == "buffer.dat" @@ -1828,7 +1838,7 @@ def test_buffer_combined_uncertainty_and_provenance_roundtrip_all_formats( ) profile = app.read_external_1d_profile(written) - np.testing.assert_allclose(profile["err_rel"], combined) + np.testing.assert_allclose(profile["err_abs"], combined) provenance = profile["operator_provenance"] assert provenance["buffer_source_name"] == "buffer.dat" assert provenance["buffer_source_sha256"] == "a" * 64 @@ -1872,7 +1882,7 @@ def test_unknown_buffer_alpha_uncertainty_stays_unknown_all_formats( ) profile = app.read_external_1d_profile(written) - assert np.all(np.isnan(profile["err_rel"])) + assert np.all(np.isnan(profile["err_abs"])) provenance = profile["operator_provenance"] assert provenance["buffer_source_name"] == "buffer.dat" assert provenance["buffer_source_sha256"] == "b" * 64 From a8c3863e3e92c94bb064f79d8f866df754e32f21 Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Sun, 16 Aug 2026 18:18:51 +0000 Subject: [PATCH 2/4] test: align remaining callers with intensity-state gates Keep column-override coverage without raw-count name conflicts, require labeled Workbench XML exports, and gate minimal_2d as K-only after thickness is already recorded. Co-authored-by: Delun Gong --- examples/minimal_2d/run_minimal_2d_pipeline.py | 5 ++++- tests/test_cli.py | 8 ++++---- tests/test_workbench_output_paths.py | 18 ++++++++++++++++++ tests/test_workbench_scientific.py | 3 ++- 4 files changed, 28 insertions(+), 6 deletions(-) diff --git a/examples/minimal_2d/run_minimal_2d_pipeline.py b/examples/minimal_2d/run_minimal_2d_pipeline.py index 3462df1..3824d29 100644 --- a/examples/minimal_2d/run_minimal_2d_pipeline.py +++ b/examples/minimal_2d/run_minimal_2d_pipeline.py @@ -167,7 +167,10 @@ def run_pipeline(output_dir: Path) -> dict[str, object]: }, } require_relative_input_for_absolute_scaling( - standard_profile, profile_name="minimal_2d standard" + standard_profile, + profile_name="minimal_2d standard", + corrections_to_apply=("k",), + required_existing_corrections=("thickness",), ) k_result = estimate_k_factor_robust( q_meas=q, diff --git a/tests/test_cli.py b/tests/test_cli.py index 689e8f7..fcab425 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -161,11 +161,11 @@ def test_cli_estimate_k_accepts_explicit_column_overrides( meas = tmp_path / "meas.csv" ref = tmp_path / "ref.csv" meas.write_text( - "angle,counts\n0.01,17.1\n0.02,15.4\n0.05,13.4\n0.10,11.8\n", + "angle,I_meas\n0.01,17.1\n0.02,15.4\n0.05,13.4\n0.10,11.8\n", encoding="utf-8", ) ref.write_text( - "q_ref,absolute\n0.01,34.2\n0.02,30.8\n0.05,26.8\n0.10,23.6\n", + "q_ref,I_ref\n0.01,34.2\n0.02,30.8\n0.05,26.8\n0.10,23.6\n", encoding="utf-8", ) @@ -182,11 +182,11 @@ def test_cli_estimate_k_accepts_explicit_column_overrides( "--q-col", "angle", "--i-col", - "counts", + "I_meas", "--ref-q-col", "q_ref", "--ref-i-col", - "absolute", + "I_ref", "--intensity-state", "relative", ], diff --git a/tests/test_workbench_output_paths.py b/tests/test_workbench_output_paths.py index 26cfe36..e80f51e 100644 --- a/tests/test_workbench_output_paths.py +++ b/tests/test_workbench_output_paths.py @@ -8,9 +8,24 @@ import numpy as np import pytest +from saxsabs.core.calibration_context import CalibrationContext from saxsabs.core.execution_policy import RunPolicy +def _labeled_export_context(): + return CalibrationContext( + formula_version="v3_nist_blank", + monitor_mode="rate", + poni_sha256="poni-sha", + mask_sha256=None, + flat_sha256=None, + correct_solid_angle=True, + polarization_factor=None, + standard_key="SRM3600", + standard_thickness_cm=0.1055, + ) + + def _load_workbench_module(): pytest.importorskip("fabio") pytest.importorskip("pyFAI") @@ -73,6 +88,9 @@ def test_workbench_save_profile_table_returns_actual_xml_path(tmp_path): np.array([0.1, 0.1, 0.1]), "Q_A^-1", output_format="cansas_xml", + calibration_context=_labeled_export_context(), + thickness_cm=0.1055, + thickness_source="certified SRM 3600 thickness", ) assert written == tmp_path / "absolute.xml" diff --git a/tests/test_workbench_scientific.py b/tests/test_workbench_scientific.py index 539b583..6103ff8 100644 --- a/tests/test_workbench_scientific.py +++ b/tests/test_workbench_scientific.py @@ -1308,7 +1308,8 @@ def test_tab3_workbench_parser_roundtrips_cansas_xml(tmp_path): assert profile["x_col"] == "Q" assert profile["i_col"] == "I" assert profile["err_col"] == "Idev" - assert profile["operator_provenance"] == {} + assert profile["operator_provenance"]["intensity_state"] == "absolute_cm^-1" + assert profile["operator_provenance"]["intensity_unit"] == "1/cm" def test_tab3_external_buffer_empty_path_fails_closed_and_updates_status(): From 2df8087f6984fc3c3f59d469a8f3b44f99026db9 Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Sun, 16 Aug 2026 18:20:18 +0000 Subject: [PATCH 3/4] docs: limit minimal_2d claims to the 9x9 homemade reducer Record that the example uses a homemade integer-bin radial average, not pyFAI or the BL19B2 campaign runner, and refuse I_abs columns in estimate-k. Co-authored-by: Delun Gong --- README.md | 12 ++++---- docs/api.md | 6 ++-- docs/reviewer-faq.md | 14 +++++----- examples/minimal_2d/README.md | 10 ++++--- .../minimal_2d/run_minimal_2d_pipeline.py | 9 ++++-- paper/paper.md | 15 +++++----- src/saxsabs/core/buffer_subtraction.py | 2 +- tests/test_cli.py | 28 +++++++++++++++++++ tests/test_minimal_2d_example.py | 6 ++++ 9 files changed, 73 insertions(+), 29 deletions(-) diff --git a/README.md b/README.md index e7b9747..db2e374 100644 --- a/README.md +++ b/README.md @@ -122,8 +122,9 @@ not experimental evidence. ## Reproducible example -The bundled example generates deterministic synthetic dark, background, -standard, and sample frames, then runs the package reduction APIs: +The bundled example plants deterministic synthetic dark, background, +standard, and sample frames on a 9×9 array, subtracts a NIST blank in detector +space, and reduces with a homemade integer-bin radial average (not pyFAI): ```bash python examples/minimal_2d/run_minimal_2d_pipeline.py @@ -141,9 +142,10 @@ and expected files. Deterministic synthetic K-factor example showing retained and rejected ratios.

-> This example recovers a planted synthetic $K$ and sample curve and checks -> labeled file content. It is not measured-beamline validation or independent -> third-party format validation. +> This example recovers a planted synthetic $K$ and sample curve from a 9×9 +> homemade radial average and checks labeled file content. It is not pyFAI +> integration, BL19B2 campaign validation, measured-beamline validation, or +> independent third-party format validation. ## Documentation diff --git a/docs/api.md b/docs/api.md index 0df9820..4d2fbf0 100644 --- a/docs/api.md +++ b/docs/api.md @@ -190,6 +190,6 @@ for their full input contract and provenance requirements. These APIs provide software operations and checks; they do not establish that a beamline measurement is calibrated. Users remain responsible for appropriate reference standards, independently measured inputs, detector geometry, valid -units, and experiment-specific acceptance. The included synthetic 2D example -recovers a planted $K$ and sample curve and checks labeled file content; it is -not experimental beamline validation. +units, and experiment-specific acceptance. The included synthetic 2D example recovers a planted $K$ and sample curve from +a 9×9 homemade radial average (not pyFAI) and checks labeled file content; it +is not experimental beamline validation. diff --git a/docs/reviewer-faq.md b/docs/reviewer-faq.md index 2187d43..4491d9b 100644 --- a/docs/reviewer-faq.md +++ b/docs/reviewer-faq.md @@ -25,13 +25,13 @@ Yes. Run: python examples/minimal_2d/run_minimal_2d_pipeline.py ``` -The script constructs independent dark, blank, SRM 3600, and sample frames, -gates the standard profile as `relative` before $K$, writes labeled -`absolute_cm^-1` outputs (CSV/TSV/canSAS XML and optional NXcanSAS HDF5), -checks that the XML exposes `i_abs` not `i_rel`, and writes numerical K and -sample-intensity errors to `summary.json`. This recovers a planted synthetic -curve within script tolerances. It is a software golden test, not measured -beamline validation or third-party format validation. +The script constructs independent dark, blank, SRM 3600, and sample frames on a +9×9 array, subtracts a NIST blank in detector space, reduces with a homemade +integer-bin radial average (not pyFAI), gates the standard profile as +`relative` before $K$, writes labeled `absolute_cm^-1` outputs with unknown +uncertainty, and records those limits in `summary.json`. It does not run the +BL19B2 campaign runner. This is a software golden test, not measured beamline +validation or third-party format validation. ## Have the structured exports been checked outside the project readers? diff --git a/examples/minimal_2d/README.md b/examples/minimal_2d/README.md index 06aa120..c08d92d 100644 --- a/examples/minimal_2d/README.md +++ b/examples/minimal_2d/README.md @@ -26,7 +26,9 @@ Expected key result: - `absolute_profile.csv`, `absolute_profile.tsv`, `absolute_profile.xml` - `absolute_profile.h5` if `h5py` is installed (`pip install -e .[hdf5]`) -The example recovers a planted synthetic $K$ and sample curve, writes labeled -`absolute_cm^-1` outputs, and checks that the XML exposes `i_abs` rather than -`i_rel`. It does not replace validation with a measured SRM 3600 coupon, -independently measured transmissions, or a third-party format consumer. +The example recovers a planted synthetic $K$ and sample curve on a 9×9 array +using a homemade integer-bin radial average (not pyFAI), writes labeled +`absolute_cm^-1` outputs with unknown uncertainty, and checks that the XML +exposes `i_abs` rather than `i_rel`. It uses `build_nist_net_image` for +detector-space blank subtraction; it does not run the BL19B2 campaign runner, +pyFAI integration, a measured SRM 3600 coupon, or a third-party format consumer. diff --git a/examples/minimal_2d/run_minimal_2d_pipeline.py b/examples/minimal_2d/run_minimal_2d_pipeline.py index 3824d29..e2bbe60 100644 --- a/examples/minimal_2d/run_minimal_2d_pipeline.py +++ b/examples/minimal_2d/run_minimal_2d_pipeline.py @@ -253,9 +253,14 @@ def run_pipeline(output_dir: Path) -> dict[str, object]: summary: dict[str, object] = { "validation_type": "deterministic_synthetic_raw_frames", "evidence": ( - "planted synthetic K and sample curve recovered within script tolerances; " - "not measured-beamline or third-party format validation" + "9x9 homemade integer-bin radial average recovers planted K and sample " + "curve; not pyFAI, not the BL19B2 campaign runner, and not measured-beamline " + "or third-party format validation" ), + "detector_shape": list(source_image.shape), + "reduction": "homemade_integer_bin_radial_average", + "uses_pyfai": False, + "uses_bl19b2_campaign": False, "points": int(q.size), "q_min": float(q.min()), "q_max": float(q.max()), diff --git a/paper/paper.md b/paper/paper.md index b158ee1..b71d486 100644 --- a/paper/paper.md +++ b/paper/paper.md @@ -147,13 +147,14 @@ Archived earlier releases are collected in the Zenodo concept record The repository includes a strict BL19B2 batch workflow from detector images to exported results. Its deterministic example in `examples/minimal_2d/` plants -synthetic dark, background, standard, and sample frames; recovers the planted -$K$ and sample curve within script tolerances; writes labeled absolute text and -canSAS XML (and NXcanSAS when `h5py` is present); and checks that the XML -exposes `i_abs`, not `i_rel`. That example is a software golden test, not -measured-beamline or third-party format validation. Automated tests cover -numerical calculations, parsers, exporters, the command-line interface, -launchers, and Workbench validation rules. +synthetic dark, background, standard, and sample frames on a 9×9 array; +subtracts a NIST blank in detector space; reduces with a homemade integer-bin +radial average (not pyFAI); recovers the planted $K$ and sample curve within +script tolerances; and writes labeled absolute text/XML with unknown +uncertainty. It does not exercise the BL19B2 campaign runner or third-party +format validation. Automated tests cover numerical calculations, parsers, +exporters, the command-line interface, launchers, and Workbench validation +rules. The repository configures continuous integration for Python 3.10--3.13 on Linux, Windows, and macOS. diff --git a/src/saxsabs/core/buffer_subtraction.py b/src/saxsabs/core/buffer_subtraction.py index 4543340..621d4dd 100644 --- a/src/saxsabs/core/buffer_subtraction.py +++ b/src/saxsabs/core/buffer_subtraction.py @@ -33,7 +33,7 @@ class BufferSubtractionResult: q : np.ndarray Momentum-transfer values (Å⁻¹). i_subtracted : np.ndarray - Subtracted intensity (cm⁻¹ or relative). + Subtracted intensity on the required absolute cm⁻¹ scale. err_subtracted : np.ndarray Propagated uncertainty. alpha : float diff --git a/tests/test_cli.py b/tests/test_cli.py index fcab425..b675f1f 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -251,6 +251,34 @@ def test_cli_parse_external1d(tmp_path: Path, capsys: pytest.CaptureFixture[str] assert out["intensity_state"] == "ambiguous" +def test_cli_estimate_k_refuses_i_abs_column_name( + tmp_path: Path, + capsys: pytest.CaptureFixture[str], + monkeypatch: pytest.MonkeyPatch, +): + meas = tmp_path / "meas.csv" + ref = tmp_path / "ref.csv" + meas.write_text( + "Q_A^-1,I_abs\n0.01,17.1\n0.02,15.4\n0.05,13.4\n0.10,11.8\n", + encoding="utf-8", + ) + ref.write_text( + "Q_A^-1,I_abs\n0.01,34.2\n0.02,30.8\n0.05,26.8\n0.10,23.6\n", + encoding="utf-8", + ) + monkeypatch.setattr( + sys, + "argv", + ["saxsabs", "estimate-k", "--meas", str(meas), "--ref", str(ref)], + ) + + with pytest.raises(SystemExit) as exc_info: + main() + + assert exc_info.value.code == 1 + assert "already absolute" in capsys.readouterr().err.lower() + + def test_cli_estimate_k_refuses_unlabeled_intensity( tmp_path: Path, capsys: pytest.CaptureFixture[str], diff --git a/tests/test_minimal_2d_example.py b/tests/test_minimal_2d_example.py index 15e6eb7..8fabf0f 100644 --- a/tests/test_minimal_2d_example.py +++ b/tests/test_minimal_2d_example.py @@ -24,7 +24,13 @@ def test_minimal_2d_example_uses_deterministic_raw_frame_golden(tmp_path: Path): assert completed.returncode == 0, completed.stderr summary = json.loads((output / "summary.json").read_text(encoding="utf-8")) assert summary["validation_type"] == "deterministic_synthetic_raw_frames" + assert summary["detector_shape"] == [9, 9] + assert summary["reduction"] == "homemade_integer_bin_radial_average" + assert summary["uses_pyfai"] is False + assert summary["uses_bl19b2_campaign"] is False assert summary["k_relative_error"] < 0.005 assert summary["sample_max_relative_error"] < 0.01 assert summary["uncertainty_status"] == "unknown_without_input_variances" + xml_text = (output / "absolute_profile.xml").read_text(encoding="utf-8") + assert "1/cm" in xml_text From a1dfeb0a28ebab750ba7cb09641776c8cf0929b5 Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Sun, 16 Aug 2026 18:20:27 +0000 Subject: [PATCH 4/4] docs: record the 9x9 homemade reducer in changelog notes Co-authored-by: Delun Gong --- CHANGELOG.md | 5 +++-- SUBMISSION_READINESS.md | 5 +++-- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 82f2926..bd53a91 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,8 +17,9 @@ - Point README and paper at the unreleased 2.0.0 tree on `main`, not GitHub Release v1.1.1. Record canSAS XSD / punx checks as offline 2026-08-15, not CI. - Tighten the paper's K-estimator, interface-equivalence, and `minimal_2d` - claims, and remove stale Draft-PR #1 merge instructions from the README and - readiness notes. + claims (9×9 homemade radial average, not pyFAI; unknown uncertainty; not the + BL19B2 campaign runner), and remove stale Draft-PR #1 merge instructions from + the README and readiness notes. ### Submission-candidate hardening (2026-08-12) diff --git a/SUBMISSION_READINESS.md b/SUBMISSION_READINESS.md index be181c0..9a48fb1 100644 --- a/SUBMISSION_READINESS.md +++ b/SUBMISSION_READINESS.md @@ -11,8 +11,9 @@ create `v2.0.0`, a GitHub Release, or a Zenodo version archive during review. count and duration are retained in the dated external validation record. - Ruff: root modules, package, tests, paper scripts, and submission gate pass. - README: 5 local images and all local links resolve; SVG/image audit passes. -- Minimal 2D example: K and sample maximum relative errors are - `0.001933697...`; CSV, TSV, XML, and HDF5 outputs are written. +- Minimal 2D example: 9×9 homemade radial average (not pyFAI) recovers planted + K and sample maximum relative errors of `0.001933697...`; CSV, TSV, XML, and + HDF5 outputs are written with unknown uncertainty. - Fresh-copy distribution build: wheel and sdist PASS from a source tree outside every Git checkout. The exact archive inventory is retained in the dated external validation record; the sdist includes README assets,