Repository navigation
Let an imaging lab name its files, and stop merging two animals into one - #20
Merged
Merged
Conversation
Three defects, found while checking what it would take to export into another standard's layout. SPECIFICATION.md §4 listed eight file types and every one was electrophysiology or behaviour. But the session vocabulary allows "calcium imaging" and "histology", the scanner recognises .isxd, .czi, .nd2, .sbx and .lif, and the README Scope claims both modalities. So a two-photon lab could scan its data and declare the session, and then have no conforming way to name the files in it -- and the BIDS export dropped them. Adds `imaging` and `histology`, and `timestamps`, which the code has always assigned and the specification never listed. Only the extensions that name one acquisition system are matched. A .tif is either an imaging stack or a slide and nothing in the file says which, so an ambiguous one keeps a word of its own rather than being assigned one of the two at random. §4 forbids a label that misdescribes, and that is the whole reason the list is short. Histology maps to BIDS `micr`, which is published. `imaging` is deliberately left unmapped: functional two-photon has no settled BIDS datatype while BEP032 is in review, so those files keep falling through the "no BIDS datatype maps to" warning, which is true, rather than claiming one no validator would accept. Second: `M-123` and `M_123` are two subjects to NDOS -- ndos_init.SUBJECT admits `-` and `_` -- and one label to BIDS, which admits alphanumerics only. The export merged them into a single subject directory and kept whichever metadata arrived first. Invisible afterwards, because the result looks exactly like a correct export of one animal. Now detected in the plan and refused before anything is written. Third: conformance rule 3 says every SessionID is unique within its subject. The checker collected session names into a dict and never read it, so the rule was never tested. It compares on a normalised form, because the filesystem already forbids two identical sibling names -- what actually happens is one recording written two ways, `2025-03-14` beside `20250314`, or a case-only difference that survives here and collapses on the next copy. A project may now fail that check having passed before. The standard did not change; the checker caught up with it. Also restores the scope sentence dropped in c156ba9. "Not replacing BIDS or NWB" is a boundary worth keeping, and losing it widened the implied claim at exactly the wrong moment. 364 -> 380 tests.
The README opened by describing a folder standard. That is the weakest thing here and the most crowded: NeuroBlueprint occupies the same ground with an institution behind it, and BEP032 is in formal review to put animal electrophysiology into BIDS proper. A lab choosing between them has no reason to pick this one, and a reviewer has no reason to call it novel. What is actually unusual is the part none of them attempt. The nearest competitor's own documentation tells labs not to retrofit existing data -- "rather than trying to re-organise existing folders and analysis code, which is always tricky" -- which leaves every drive that already exists unserved. That is the drive this was written for. So the README now opens on recovery, discovery and traceability, and the specification is introduced as one component, separable from the rest, for labs that want it. Same for the package description and the citation abstract, which both led with "a BIDS-inspired standard". The status note now says plainly that no external lab has completed a pilot. That was true before and unstated, which is the kind of omission that costs more than it saves. No code change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three defects, found while working out what it would take to export into another standard's layout. All three are in a tool whose pitch is data integrity.
A two-photon lab had no way to name its files
SPECIFICATION.md§4 listed eight file types. Every one was electrophysiology or behaviour.But
session_typeallowscalcium imagingandhistology, the scanner recognises.isxd,.czi,.nd2,.sbx,.lif, and the README Scope claims both modalities. So a lab could scan its data, declare a calcium-imaging session, and then find no conforming way to name the files in it — andndos convert bidsdropped them.Adds
imagingandhistology, plustimestamps, which the code has always assigned and the spec never listed. The test that documents types against the spec had an exemption fortimestamps; that exemption is gone, which is what would have caught this.Only extensions that name one acquisition system are matched. A
.tifis either an imaging stack or a slide, and nothing in the file says which, so an ambiguous one keeps a word of its own rather than being assigned one of the two at random. §4 forbids a label that misdescribes — that rule is the reason the list stays short.Histology maps to BIDS
micr.imagingis deliberately left unmapped: functional two-photon has no settled BIDS datatype while BEP032 is in review, so those files keep falling through the existing warning, which is true, rather than claiming a datatype no validator would accept.Two animals were being merged into one
ndos_init.SUBJECTadmits-and_, soM-123andM_123are two subjects. BIDS labels admit alphanumerics only, so both becomeM123.The export merged them into one subject directory and kept whichever metadata arrived first. Invisible afterwards — the result is indistinguishable from a correct export of a single animal. Detected in the plan now, and refused before anything is written.
Conformance rule 3 was never enforced
"Every
SessionIDis unique within its subject." The checker collected session names into a dict and never read it back.It compares on a normalised form, because a literal comparison can never fire — the filesystem already forbids two identical sibling names. What actually happens is one recording written two ways,
2025-03-14beside20250314, or a case-only difference that survives here and collapses on the next copy.A project may now fail this check having passed before. The standard did not change; the checker caught up with it. Noted in the spec's versioning section.
Also
Restores the scope sentence dropped in
c156ba9. "Not replacing BIDS for human MRI/MEG/EEG, replacing NWB as a container" is a boundary worth keeping — losing it widened the implied claim.No version bump: §4 is a SHOULD and the additions are optional, so a project that conformed before still conforms. The spec's own rule only requires a new number when a conforming project would stop conforming.
364 → 380 tests.