Tracking: format-conformance review of brukerapi against FILE_FORMAT.md (PRs #156-#165)

[gdevenyi]

Umbrella issue for a format-conformance and data-loading review of brukerapi
against FILE_FORMAT.md,
and for the ten pull requests that act on it.

Method. Every finding was reproduced against a corpus of 3,468 binaries
(34 GB) spanning PV5.1 / PV6.0.1 / PV7.0.0 / PV360 3.x, and each fix was
re-measured over the same corpus. Where the specification alone could not settle
a question, an independent oracle was used: the vendor 2dseq for fid slice
ordering and EPI trimming, VisuCorePosition for geometry, cross-plane
intensity profiles for frame transposition, orthonormality of ACQ_grad_matrix
for parameter reshape order. The findings that no dataset reaches are labelled
as such and are reproduced synthetically.

Baseline, and what it stays at. 3,197 of 3,468 targets load; the 271
non-loads are .npz sidecars, TopSpin ser, LFS stubs and empty
reconstructions, none of them a defect. That number is unchanged by all ten PRs.
Across the whole series, exactly 22 datasets change shape — the 2 EPSI fids that
were dropping 75 % of their samples, and the 20 field maps that gain the
singleton repetition axis they always had.

Pull requests

They are a stack, in this order; each is one commit and stands on its own, but
the diffs contain the earlier commits until those merge.

#	PR	Findings	Severity
1	#156 Derive the 2dseq affine from VisuCorePosition/VisuCoreOrientation	G1–G4	blocker
2	#157 Join wrapped JCAMP-DX lines by deleting the break, and stop write() corrupting records	J1, J2	blocker
3	#158 Parse escaped delimiters and nested structs instead of discarding them	J3, J4, J5	high
4	#159 Make FrameGroupSplitter.write() work and honour add_parameters=	A1, A2	high
5	#160 Read every acquired sample of an EPSI fid	R1	high
6	#161 Read transposed frames in their stored shape and own frame-group metadata by descriptor	D1, D2	high
7	#162 Reach PV360 raw data through Study, and read traj from an archive	A3, R4	high/medium
8	#163 Label the acquisition-object axis for what it is, and keep slice distance a double	R2, R5, R6, D4	medium
9	#164 Fix the API surface: metadata groups, reports, CLI options, version gates	A5–A9, D9, D10, R7, R9, J6, J7	medium/low
10	#165 Close the spec paths the corpus never exercises	D3, D5–D8, R3, R8, J8	latent

The headline is #156: Dataset.affine was numerically wrong on every image
dataset in the corpus — median 35 mm, maximum 126 mm — and mirrored in x and y
for 87 % of them, while 35 spectroscopy datasets received a plausible identity
matrix. After it, affine @ (0,0,0,1) == VisuCorePosition[0] holds on
1,591/1,591 datasets, no affine is singular (23 were), and non-spatial frames
refuse instead of inventing geometry.

Second is #157: Dataset.write() was not a fixed point — over a 400-file
sample, 289 files came back with a corrupted parameter and only 108 survived a
re-write unchanged. Afterwards all 400 round-trip with every record identical
and 269 are byte-identical to the vendor file.

Testing

Each PR adds tests that fail without its change. They are synthetic: a new
test/synthetic.py writes minimal ParaVision datasets from the parameter shapes
of real files, so the suite exercises geometry, frame groups, scaling, slice
packages, raw layouts and archives without any vendor data being present.
python -m pytest test goes from 158 to 226 passing. The four failures that
remain reference the legacy test/test_data/PV601|PV700 layout and predate this
work.

Functional gaps (review §6.2)

Spec areas the library does not implement, as opposed to implements wrongly.
Each has its own issue with a concrete proposal:

Spec	Gap	Status
§7.2, §12	image affine from VisuCorePosition/VisuCoreOrientation	closed by #156
§5.4, §12	raw acquisitions carry no geometry at all	#166
§12	VisuCoreAtsCenterDistance — ATS cradle origin	#167
§5.3, §6.2	ACQ_scan_size / RECO_inp_order inferred from scheme_id	#168
§7.10	multi-package splitting needs parameters PV5.1 never writes	#169
§3.3, §13.1	STORE_discard, nStoredScans cross-check	#170 (storageDataType closed by #165)
§13	methreco / pvmeta unreachable	closed by #164

Notes and open questions

• EPSI spectral interleave (Read every acquired sample of an EPSI fid (R1) #160): that ¾ of the samples were dropped, and
  that the target is 96×256×64, are both certain; whether the spectral index
  runs spectral·NSegments + segment or the reverse is not, and wants a dataset
  with a known spectrum.
• Square transposed frames (Read transposed frames in their stored shape and own frame-group metadata by descriptor (D1, D2) #161): the exchange is applied only when the two
  dimensions differ in length, because square frames measure as already
  consistent with VisuCoreOrientation (0.99 as delivered, −0.05 once swapped).
  Whether Bruker's own export presents them with row-swapped orientation
  matrices instead is unresolved.
• Pixel values were not validated against an independent implementation.
  Slice ordering, frame-group assignment, EPI trimming and k-space decoding were
  cross-validated against the vendor 2dseq; absolute intensities, phase
  conventions and channel ordering were not.
• The affine was validated against VisuCorePosition, the family of
  parameters it derives from. That proves the old derivation inconsistent with
  §7.2/§12 and the new one self-consistent, and it is exact on 54/57 of two
  purpose-built orientation phantom collections — but a ParaVision DICOM export
  would settle it independently.

[gdevenyi]

All ten PRs are green on CI, including the full test job, which runs against the downloaded Zenodo and PV360 data.

Verified locally against the same data as well, per branch, by symlinking the corpus into the layout conftest expects (test/test_data/PV51|PV601|PV700|PV360_StdData) so that the ~1,900 corpus-parametrized tests actually collect:

branch	result
#156 affine	2053 passed, 12 skipped
#157 jcampdx joins/write	2059 passed
#158 escapes/nested structs	2062 passed
#159 splitter/add_parameters	2066 passed
#160 EPSI	2069 passed
#161 transposition/frame groups	2073 passed
#162 PV360 study/archive traj	2076 passed
#163 axis labels	2079 passed
#164 API surface	2093 passed
#165 latent spec paths	2107 passed

No failures at any point in the stack; the count grows only because each PR adds its own tests.

Two things CI found that a corpus-only sweep could not, both now handled in the PR that caused them:

• the shipped .npz caches for the two EPSI fids were written by the pre-fix reader, so they hold a fraction of the samples the corrected layout returns (Read every acquired sample of an EPSI fid (R1) #160 documents that and asserts the cached values are still present);
• three tests assert axis labels or companion shapes that Label the acquisition-object axis for what it is, and keep slice distance a double (R2, R5, R6, D4) #163/Fix the API surface: metadata groups, reports, CLI options, version gates (A5-A9, D9, D10, R7, R9, J6, J7) #164/Close the spec paths the corpus never exercises (D3, D5-D8, R3, R8, J8) #165 deliberately change, and they only run when the downloaded data is present.