Skip to content

Read every ASM measurement and its peaks; write peaks in their unit; release 1.5.5 - #81

Merged
ekwan merged 26 commits into
mainfrom
asm-read-wide
Oct 17, 2026
Merged

ekwan merged 26 commits into
mainfrom
asm-read-wide

Conversation

@ekwan

@ekwan ekwan commented Oct 17, 2026 •

Copy link
Copy Markdown
Collaborator

rb.from_asm now reads every measurement's peaks and rebuilds every chromatogram cube, to_asm writes each peak in its channel's unit, and there are public readers for one injection's peaks, one ASM measurement's peaks, the vendor and the technique. Release 1.5.5.

  • Real data: 531 local Empower and OpenLab exports hold 20,026 peaks. 1.5.4 read 15,503 of them (5,356 in each channel's first integration). This branch reads all 20,026 (6,769 first). The missing ones were the fluorescence channels' 4,521 peaks and an FID run's 2. Re-exported fluorescence files validate against REC/2026/06.
  • Peaks without a trace: peaks are read from every measurement, including one whose cube has no values (an index document). Each group also carries the channel's detector and metadata.
  • Non-UV traces: each chromatogram cube is rebuilt with metadata["unit"] as written. The detector comes from the device type, then the label or identifier, then the measure: UV, RID, FLD (new), ELSD, FID or CAD. A channel rainbow has no detector name for (conductivity) is detector None and goes in analog. The metadata also holds measure, device_type, acq_method, wavelength, bandwidth, reference_wavelength (None for Ref=off), reference_bandwidth, excitation_wavelength and emission_wavelength.
  • Peaks: area_unit and height_unit come as written for non-absorbance (RFU, pA). They're None only when the document gives no unit. Absorbance still converts to mAU. Each peak also has identifier and index, and each integration's processing has description.
  • Envelope: injection_time, sample_role, well_plate_id, acq_method, and the data_system, analysis_sequence and project records. A scalar integer datum is read as an int.
  • Public: rb.read(path, peaks=True) finds peaks the way read_sequence does. Also new: rb.peaks_from_asm(measurement), rb.sniff_vendor(path) and rb.asm_technique(run) (returns "LC" or "GC").
  • Fixed: to_asm wrote every peak area as mAU.s. Now it writes the cube's unit (pA.s for FID and CAD). REC/2026/06 has peak lists for pA, RLU, mV and RFU, so these channels keep their own cube. Only nA and V are still relabeled to absorbance. The relabel note now also records the real concept and unit as custom fields, so from_asm reads a relabeled channel back in its real unit, including a 1.5.4 export that has only the note's sentence.
  • Changed: Ref=off now reads as reference_wavelength: None in the vendor readers too. Three tests/outputs files gain the key.
  • Warnings: to_asm warns once for each peak group it can't carry (index-document peaks, analog channels), and rb.read(..., peaks=True) warns for a .dx archive as it does for Waters.
  • Tests: 819 tests pass, including 19 schema tests run with RAINBOW_TEST_ASM_SCHEMA=1. New tests cover each feature.

🤖 Generated with Claude Code

https://claude.ai/code/session_01BZZc1jNciuVx7aekxG7YRR

Eugene Kwan and others added 26 commits October 17, 2026 10:52
parse_optics gave a Ref=off channel no reference keys at all, the same as a
description that says nothing about a reference. The two are different
statements: "DAD1A,Sig=210,4 Ref=off" and "DAD1A,Sig=210,4 Ref=360,100" are
two channels at one wavelength, and a reader matching channels by their
optics could not tell the first from one whose reference was simply not
recorded. Ref=off now reads as reference_wavelength None, present, with no
reference_bandwidth, in the Chemstation, OpenLab and MassHunter DAD reads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BZZc1jNciuVx7aekxG7YRR
to_asm wrote every peak's area as mAU.s and its height as mAU, whatever
the channel measured, so an FID or CAD peak table, whose values are pA.s
and pA, went out under a unit it was not in. The relabel that made that
look consistent, a non-absorbance channel with peaks rewritten as an
absorbance cube, existed because the schema once modelled peak lists on
absorbance only. REC/2026/06 gives electric current in pA, intensity in
RLU, voltage in mV, fluorescence in RFU and counts their own peak lists,
each pinned to the measure's unit.

A channel in one of those measures now keeps its faithful cube when it
carries peaks, and its peaks are written in the cube's unit: area in
unit.s, height and edge heights in unit. A peak that names its own unit
(area_unit, height_unit) keeps it. Only a measure with no peak list in
the schema (nA, V) is still relabeled, with its note. The FID, CAD,
nA-relabel and fluorescence exports validate against the published LC
and GC schemas.

FLD is a detector kind now, so a fluorescence channel exports as a
fluorescence cube in RFU, with its excitation and emission wavelengths
as device control settings. A peak's identifier is written when present.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BZZc1jNciuVx7aekxG7YRR
from_asm and sequence_from_asm read a measurement's peaks only when they
had rebuilt its trace, and they rebuilt only absorbance cubes with inline
data. Everything else was dropped with its peaks: every fluorescence
channel of an Empower study, an FID run, and every measurement of an
index document, whose cubes carry their structure and none of their
values. Over 531 local ASM files, 1.5.4 read 15,503 of 20,026 peaks
(5,356 in each channel's first integration); this reads all 20,026.

Peaks are now read from every measurement, whatever its cube measures
and whether or not the trace could be rebuilt. Each group also carries
the channel's detector and metadata, so a group with no trace beside it
still says which channel it was integrated on.

Every chromatogram cube with inline values is rebuilt, with its unit as
written (metadata["unit"]) and its measure concept (metadata["measure"]).
The detector comes from the device type, then the cube label or
measurement identifier, then the measure: UV, RID, FLD, ELSD, FID, or CAD;
a channel rainbow has no name for (a conductivity detector) is detector
None and joins the directory's analog traces. The device type and the
device acquisition method are read too. The optics come from the device
control settings, then the cube label, then the processed data
description: wavelength, bandwidth, the reference (Ref=off as None),
and a fluorescence channel's excitation and emission.

A peak's height and area convert to mAU only from an absorbance unit;
any other unit (RFU, pA) is kept as written, in height_unit and
area_unit, None only when the document wrote none. A peak's identifier
and @index are read, and each integration's processing records the
processed data description.

The envelope gains injection_time, sample_role and well_plate_id (the
last two written back), acq_method, and the data system, analysis
sequence and project records, field by field as written. A scalar
integer datum reads as an int.

rainbow.asm.peaks_from_asm reads one measurement document's peaks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BZZc1jNciuVx7aekxG7YRR
…ique

Only read_sequence(peaks=True) attached a run's integrated peaks, so a
caller holding one .D had to reach into rainbow.agilent for the private
helpers read_sequence uses. rb.read(path, peaks=True) now attaches them
to the one injection, found the way read_sequence finds them: from the
sequence.acaml beside the .D, matched by its folder name, else from the
.D's own sequence.acam_. rainbow.agilent.injection_peaks(path) is the
same lookup on its own. A Waters run read with peaks=True warns that its
results are not read and has an empty list.

rb.peaks_from_asm reads one ASM measurement document's peaks, the reader
from_asm uses for each measurement. rb.sniff_vendor is the content check
rb.read falls back to for a folder without a vendor suffix (the private
name is kept), and rb.technique(run) is the "LC" or "GC" the export will
use, so callers stop reaching for _sniff_vendor and asm._technique.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BZZc1jNciuVx7aekxG7YRR
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BZZc1jNciuVx7aekxG7YRR
A CAD channel in nA with peaks is exported relabeled to absorbance in mAU,
and from_asm read it back as a CAD channel in mAU with peaks in mAU.s. The
next to_asm then wrote an electric current cube with mAU.s peaks, which
the schema rejects. A 1.5.4 export of a pA channel has the same shape.

The relabel note now records the real measure as two custom fields,
reported measure concept and reported measure unit, beside the sentence.
from_asm reads them, or the sentence alone in a 1.5.4 document, and
returns the channel in its real measure and unit and its peaks in that
unit (the relabel never converted the values). to_asm writes a peak's
units as the cube's, the only ones the schema admits beside it, and warns
when a peak stated another. Both an export, read, export lap and a 1.5.4
relabel validate on the second export.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BZZc1jNciuVx7aekxG7YRR
to_asm dropped a peak group no exported measurement carries without a
word: peaks read from an index document, whose cubes had no values, or
on an analog channel rainbow has no detector name for. It now warns once
per such group, and the round-trip guide lists both cases.

rb.read(path, peaks=True) on an OpenLab .dx archive returned an empty
list as if nobody had integrated the run; it now warns that the
archive's results are not read, as it does for Waters.

A channel whose unit is already the one its detector publishes (an FLD
channel's RFU, an ELSD's RLU) no longer warns that it was relabeled.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BZZc1jNciuVx7aekxG7YRR
technique alone reads as a property of the run in general; the function
answers which ASM document the export writes. The changelog and the
detector guide name Counts among the measures with their own peak list,
and the 1.5.5 entry records the relabel round trip and the warnings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BZZc1jNciuVx7aekxG7YRR
"FLD1 A, Ex=230, Em=310" has no Sig= clause, so the 130/30 container's
lenient reading took its first number for a UV wavelength and typed the
channel UV at 230 nm. A signal that names an excitation and an emission is
now an FLD channel with excitation_wavelength and emission_wavelength in
its metadata, labelled by its emission as from_asm labels one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BZZc1jNciuVx7aekxG7YRR
- A peak's µV (and AU) converts to mAU only on an absorbance channel. On
  an ELSD channel in µV a 3000 µV peak read back as 3.0 mAU beside a
  trace still in µV. The measure is the cube's, or a relabeled channel's
  reported one.
- Agilent's "Ex=230, Em=310" label spelling is read.
- A fluorescence detector's wavelength setting is its emission and
  outranks the label's, as every other setting does.
- The well plate identifier is written as the sample's custom
  information, since REC/2026/06 has no sample document property for it,
  and read back from there into well_plate_id.
- _PEAK_MEASURES lists every measure the LC and GC schemas give a peak
  list: charge in nC, angle in m°, conductance in S, conductivity in S/m
  join the others, and the detector guide says so.
- A relabeled channel no longer warns a second time, per peak unit, on
  re-export.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BZZc1jNciuVx7aekxG7YRR
A .dx (or Waters) run the vendor reader cannot read returns None, and
attaching peaks to it raised AttributeError instead of the unreadable
error. injection_peaks says read_sequence(peaks=True) is the efficient
path for many injections, since each call parses the whole result
document. The asm_technique test asserts the sequence's actual
technique. The changelog records the µV and FLD changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BZZc1jNciuVx7aekxG7YRR
Only the ChemStation .ch reader typed "FLD1 A, Ex=230, Em=310" as FLD.
parse_optics now reads the Ex=/Em= clause into excitation_wavelength and
emission_wavelength, and the readers that share it agree: the .ch reader
uses it, OpenLab's _classify types a trace FLD by its device or that
clause before it tests for absorbance (and labels it by its emission
whatever the payload's header says), and the MassHunter reader takes an
FLD's .cd/.cg pair, which is the DAD's format, reading a signal that names
an excitation and emission as FLD. An FLD's spectra are not read as UV.
Tested on a ChemStation file and a .dx built from the fixtures, and a
synthetic MassHunter pair.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BZZc1jNciuVx7aekxG7YRR
- The processing time is written in the data processing document, where
  REC/2026/06 puts it, and the 2025/03 processed data description as the
  processed data document's custom information; both are read from either
  place. A schema test checks every name rainbow writes in those two
  documents against the names the schema declares, since the embed
  schemas are open and would not catch one.
- from_asm scales an absorbance trace in AU (or Empower's µV) to mAU, as
  its peaks already were, so trace minus edge height is right; the unit in
  its metadata says mAU.
- Each measure with a peak list maps to the area unit its schema names,
  [S/m].s for conductivity rather than S/m.s.
- The asm_technique test states its expectation; the peak-unit warning
  names its channel; excitation_bandwidth is documented.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BZZc1jNciuVx7aekxG7YRR
A peak read from a result document now carries identifier, index,
area_unit and height_unit as None, so a vendor peak and one read from ASM
have one shape. read()'s peaks argument comes before the private
_labels_only in rb.read and rainbow.agilent.read (every caller passes
_labels_only by name), injection_peaks takes an absolute path so a run
read by a relative name still finds the sequence.acaml beside it, and the
detector list in DataDirectory's docstring names FLD.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BZZc1jNciuVx7aekxG7YRR
- data processing time goes through the timestamp routine and takes
  utc_offset=; one that still has no offset is the processed data
  document's custom information, since the data processing document needs
  a dateTimeStamp. The schema test covers an offset-less time.
- An FLD channel read out in mV, V, pA or nA keeps its emission: the
  wavelength setting is renamed for the FLD detector or an excitation
  setting, not only for a fluorescence cube.
- _PEAK_MEASURES and the detector guide list only the measures rainbow
  exports a chromatogram in; the schema's others are named as such.
- A sample role outside the schema's enumeration is written as custom
  information, and read back into sample_role.
- injection_time is written back (in the injection document, or as the
  measurement's custom information where an LC run has none), and the data
  system, analysis sequence and project records beside the device system,
  their other fields as custom information. A record missing a required
  field is not written, and said.
- tests/inputs/rainbow_1_5_4.asm.json was written by rainbow 1.5.4
  (ff716a7) from red.D's CAD and DAD1B channels with three peaks each; it
  reads back in pA and re-exports to a valid document.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BZZc1jNciuVx7aekxG7YRR
ChemStation's parse_optics and from_asm's label reader each carried their
own Ex/Em pattern. rainbow._optics.excitation_emission is now the one
parser: the Agilent readers (through parse_optics), the Waters _CHRO
reader and from_asm all ask it. A Waters channel named like "ACQUITY FLR
ChA Ex280,Em340 nm" is now FLD with its excitation and emission,
labelled by its emission, where it was an analog trace.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BZZc1jNciuVx7aekxG7YRR
- A re-exported data system document names rainbow-api and its version
  as converter, now as conversion time and the file written as file
  identifier (each per-injection file its own). The fields naming the
  data's source are kept; the source converter's go under custom
  information, prefixed "source". Record times take utc_offset=.
- An integration's description is written as REC/2026/06's description
  and read from it, the 2025/03 processed data description, or custom
  information.
- A vendor peak and its group now have exactly the keys of one read from
  ASM: the edge heights and baseline keys as None, the group's detector,
  metadata and processing as None and alternates empty.
- The detector is not read from a measurement identifier, a key whose
  GUID segment (adc4) typed a channel CAD.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BZZc1jNciuVx7aekxG7YRR
A result document names the channel a signal was integrated on but not
its detector, so every group read with peaks=True had detector None.
rb.read(peaks=True) and read_sequence(peaks=True) now take each group's
detector and metadata from the run's DataFile for that channel.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BZZc1jNciuVx7aekxG7YRR
- A CAD's device control names it as its device identifier, beside the
  generic class, and from_asm reads the device identifier after the
  device type: rainbow's own CAD (red.D, blue.raw) reads back as CAD and
  re-exports, where blue.raw's came back an analog trace.
- The acquisition method is written as the measurement's custom
  information (device acquisition method), REC/2026/06 having no device
  control property for it, and read from either place.
- A UV channel's bandwidth and reference go to the schema's settings,
  Ref=off as the device control's custom information, and an FLD's
  excitation bandwidth and bandwidth too; electronic absorbance
  wavelength setting is read as the wavelength.
- A schema test checks the names in the sample, device control, injection
  and record documents against the schema's, following $refs.
- Docs: the detector is not named from the measurement identifier; the
  OpenLab, MassHunter and Waters readers produce FLD.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BZZc1jNciuVx7aekxG7YRR
- A channel published under its detector's default unit because the
  schema has no term for its own (red.D's CAD in mAu, as pA) carries the
  structured reported-measure note, as a peak relabel does, and from_asm
  restores its unit and its peaks'. The note keeps what was already in
  the measurement's custom information.
- rb.peaks_from_asm raises a ValueError for a whole ASM document.
- rainbow.__version__ (rainbow/_version.py) is the version an export
  names as its converter, not the installed metadata an editable install
  leaves stale; a test checks it against pyproject.toml.
- A peak area per minute is converted to per second; an absorbance area
  per an unknown time reads as None, rather than going out 60x off.
- A single-wavelength UV channel's reference wavelength is device control
  custom information: REC/2026/06 defines that setting for the spectrum
  detector only. Found by a schema test that checks each device control
  against its own detector variant (with an FLD channel and teal.dx
  added), not the union of all.
- A record's custom information is read back, so the source converter's
  fields survive a second round trip, its conversion time as a timestamp
  datum.
- CHANGELOG: the cube rebuild is listed under Changed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BZZc1jNciuVx7aekxG7YRR
The docs and docstrings called a whole sequence "a run" in places
("DataSequence (a whole run)", "the run-level files", "what the run
knows that one injection does not"), while everywhere else a run is one
injection. A sequence is now the whole batch and a run one injection, in
the sequence guide, the ASM pages and the docstrings. No API name
changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BZZc1jNciuVx7aekxG7YRR
- An FLD read out in mV, V or pA is a voltage or current cube, whose
  detector variant declares no excitation setting; its excitation
  wavelength and bandwidth go to the device control's custom information
  there (properties only on a fluorescence cube), and read back. The
  variant schema test now carries an FLD in mV.
- A vendor peak takes its height_unit and area_unit (unit + ".s") from
  the channel's DataFile when peaks are attached.
- _absorb_records steps past a non-dict chromatography document rather
  than stopping at it.
- ChemStation types a signal starting FLD<n> as FLD even when its Ex/Em
  are not numbers, rather than falling through to FID.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BZZc1jNciuVx7aekxG7YRR
- Beside an absorbance cube, which is written in mAU, a peak stated in
  AU, mAU or Empower's µV (per second or per minute for an area) is
  converted as the cube is, through _peak_measure; only a unit it cannot
  convert is relabeled, and said. It was relabeled 1000x off.
- _channel_metadata normalizes an absorbance unit to mAU, so a peak
  group's metadata says what its DataFile and peaks do.
- _detector_of stops at a device type it recognizes but has no name for
  (thermal conductivity, electron capture, mass spectrometer) rather than
  naming the channel from its label.
- MassHunter: in an FLD file, a signal naming Ex or Em in any form, or in
  LU, is a fluorescence signal, not telemetry.
- from_asm and sequence_from_asm record the document's technique in
  metadata["technique"], which the export follows: a gas chromatography
  document without an FID came back as liquid chromatography.
- Docs: the ASM file identifier is the file written, else one named for
  the run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BZZc1jNciuVx7aekxG7YRR
…ifier

- A data system record read back from an earlier re-export keeps its
  source fields as they were: the converter fields just popped are that
  export's (rainbow's) and are dropped, so a source with no conversion
  time no longer gains rainbow's, and exports 2 and 3 agree apart from
  the conversion time (checked on the gc_peaks sweep fixture too).
- A record naming both a file and a database row keeps the row and
  carries the UNC path and file name as custom information, rather than
  being dropped whole.
- Every detector's device control names it as its device identifier, so
  an ELSD in a forced gas chromatography document reads back as ELSD.
- Docs: the conversion time is the export's, so two exports differ there;
  a 1.5.4 Waters CAD export reads back as analog; the FLD fix also covers
  the 179/181 .ch container, where 1.5.4 typed FLD as FID.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BZZc1jNciuVx7aekxG7YRR
- A UV channel in µV is published as mAU unconverted, under a note; read
  back, its trace was scaled by 0.001 while its peaks stayed in µV. The
  trace's scaling now comes from the cube's own unit only: a unit from a
  relabel note is the one the values are in, and trace, peaks and
  metadata all stay in it. Tested over two laps with the apex minus the
  start height.
- A generic CAD/ADC match in a device identifier or label no longer names
  an absorbance, fluorescence or refractive index channel CAD.
- A peak area per minute is converted to per second beside any cube, not
  only an absorbance one.
- _absorbance_unit_key reads a product written with *, a middle dot or a
  space before the time, and sec as s ("µV*sec" is "µv.s").
- The source ASM conversion time is kept exactly as written, as a
  timestamp datum, across round trips.
- CHANGELOG: every chromatogram channel's device control.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BZZc1jNciuVx7aekxG7YRR
…source

- A channel's device identifier is the source's again, read into
  metadata["device_identifier"] and written back, so it still names the
  module the device system document lists (Empower's "DAD", "ACQ-FLR").
  rainbow's own detector name goes to the device control's custom
  information as "rainbow detector", which _detector_of reads first, and
  is the device identifier only where the source had none.
- A data system record whose converter is rainbow-api is rainbow's own:
  its converter fields are never promoted to source fields, so a source
  with no ASM fields gains none on a second round trip.
- CHANGELOG: reference_wavelength may now be present with the value None.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BZZc1jNciuVx7aekxG7YRR
@ekwan
ekwan merged commit dd8b8ad into main Oct 17, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant