diff --git a/source/concepts.md b/source/concepts.md index 7ccc685..b7ae063 100644 --- a/source/concepts.md +++ b/source/concepts.md @@ -1,10 +1,10 @@ # Core Concepts -This page explains how the TRX format represents tractography data on disk and -in memory. +This page describes the TRX file format specification. The authoritative (and dry!) text of the specification is maintained in the [trx-spec](https://github.com/tee-ar-ex/trx-spec) repository, and can be read [there](https://tee-ar-ex.github.io/trx-spec). To learn more about the motivation for TRX, refer to {doc}`design`. ## The TRX format +### Layout A TRX file is a ZIP archive (or on-disk directory) whose layout directly encodes its data model: @@ -19,13 +19,28 @@ encodes its data model: Coordinates are stored in **RAS+ world space** (millimeters), matching the convention used by MRtrix3 `.tck` and NIfTI qform outputs. -## Positions array +### Header + +`header.json` stores: + +- `VOXEL_TO_RASMM` — 4x4 affine mapping voxel indices to RAS+ world + coordinates (mm) +- `DIMENSIONS` — reference image grid dimensions as three `uint16` values +- `NB_STREAMLINES` — number of streamlines (`uint32`) +- `NB_VERTICES` — total number of vertices across all streamlines (`uint64`) + +The header is primarily for human readability and downstream compatibility. The +authoritative sizes come from the array dimensions themselves. + +### Arrays + +#### Positions array All streamline vertices are stored in a single flat matrix of shape `(NB_VERTICES, 3)`. Keeping all vertices contiguous enables efficient memory mapping and avoids per-streamline allocations. -## Offsets and the sentinel +#### Offsets and the sentinel `offsets` is a prefix-sum index of length `NB_STREAMLINES + 1`. Element *i* is the offset in `positions` of the first vertex of streamline *i*. The final @@ -39,7 +54,7 @@ length_i = offsets[i + 1] - offsets[i] This design avoids per-streamline allocations and makes slicing the global positions array fast and uniform. -## Data per vertex (DPV) +#### Data per vertex (DPV) A DPV array stores one value per vertex in `positions`. It has shape `(NB_VERTICES, 1)` for scalar fields or `(NB_VERTICES, N)` for vector-valued @@ -52,7 +67,7 @@ fields. Typical uses: DPV arrays live under `dpv/` and are memory-mapped in the same way as `positions`. -## Data per streamline (DPS) +#### Data per streamline (DPS) A DPS array stores one value per streamline. It has shape `(NB_STREAMLINES, 1)` or `(NB_STREAMLINES, N)`. Typical uses: @@ -65,7 +80,7 @@ DPS arrays live under `dps/` and are loaded as typed matrix fields. If the on-disk dtype differs from the requested typed reader dtype, values are converted during load. -## Groups +### Groups A group is a named list of streamline indices stored as a `uint32` array under `groups/`. Groups enable sparse, overlapping labeling: a streamline can belong @@ -75,7 +90,7 @@ to multiple groups, and groups can have different sizes. Typical uses: - Cluster assignments from QuickBundles or similar algorithms - Connectivity subsets (streamlines connecting two ROIs) -## Data per group (DPG) +#### Data per group (DPG) DPG attaches metadata to a group. Each group folder `dpg//` can contain any number of scalar or vector arrays. Typical uses: @@ -84,15 +99,106 @@ any number of scalar or vector arrays. Typical uses: - Per-bundle display color - Volume or surface-area estimates -## Header +### Accepted datatypes -`header.json` stores: +- int8, int16, int32, int64 +- uint8, uint16, uint32, uint64 +- float16, float32, float64 +- bit (for binary/boolean arrays) -- `VOXEL_TO_RASMM` — 4x4 affine mapping voxel indices to RAS+ world - coordinates (mm) -- `DIMENSIONS` — reference image grid dimensions as three `uint16` values -- `NB_STREAMLINES` — number of streamlines (`uint32`) -- `NB_VERTICES` — total number of vertices across all streamlines (`uint64`) +## Example structure -The header is primarily for human readability and downstream compatibility. The -authoritative sizes come from the array dimensions themselves. +``` +OHBM_demo.trx +├── dpg +│ ├── AF_L +│ │ ├── mean_fa.float16 +│ │ ├── shuffle_colors.3.uint8 +│ │ └── volume.uint32 +│ ├── AF_R +│ │ ├── mean_fa.float16 +│ │ ├── shuffle_colors.3.uint8 +│ │ └── volume.uint32 +│ ├── CC +│ │ ├── mean_fa.float16 +│ │ ├── shuffle_colors.3.uint8 +│ │ └── volume.uint32 +│ ├── CST_L +│ │ └── shuffle_colors.3.uint8 +│ ├── CST_R +│ │ └── shuffle_colors.3.uint8 +│ ├── SLF_L +│ │ ├── mean_fa.float16 +│ │ ├── shuffle_colors.3.uint8 +│ │ └── volume.uint32 +│ └── SLF_R +│ ├── mean_fa.float16 +│ ├── shuffle_colors.3.uint8 +│ └── volume.uint32 +├── dpv +│ ├── color_x.uint8 +│ ├── color_y.uint8 +│ ├── color_z.uint8 +│ └── fa.float16 +├── dps +│ ├── algo.uint8 +│ ├── algo.json +│ ├── clusters_QB.uint16 +│ ├── commit_colors.3.uint8 +│ └── commit_weights.float32 +├── groups +│ ├── AF_L.uint32 +│ ├── AF_R.uint32 +│ ├── CC.uint32 +│ ├── CST_L.uint32 +│ ├── CST_R.uint32 +│ ├── SLF_L.uint32 +│ └── SLF_R.uint32 +├── header.json +├── offsets.uint64 +└── positions.3.float16 +``` + +## Example code + +For a much more extensive example, please see {doc}`showcase` or visit the +examples provided in each of the language-specific implementations (linked in the +sidebar to the left). + +```python +from trx.trx_file_memmap import TrxFile, load, save +import numpy as np + +trx = load('complete_tractogram.trx') + +# Access the header (dict) / streamlines (ArraySequences) +trx.header +trx.streamlines + +# Access the dpv (dict) / dps (dict) +trx.data_per_vertex +trx.data_per_streamline + +# Access the groups (dict) / dpg (dict) +trx.groups +trx.data_per_group + +# Get a random subset of 10000 streamlines +indices = np.arange(len(trx.streamlines._lengths)) +np.random.shuffle(indices) +sub_trx = trx.select(indices[0:10000]) +save(sub_trx, 'random_1000.trx') + +# Get sub-groups only, from the random subset +for key in sub_trx.groups.keys(): + group_trx = sub_trx.get_group(key) + save(group_trx, '{}.trx'.format(key)) + +# Pre-allocate memmaps and append 100x the random subset +alloc_trx = TrxFile(nb_streamlines=1500000, nb_vertices=500000000, init_as=trx) +for i in range(100): + alloc_trx.append(sub_trx) + +# Resize to remove the unused portion of the memmap +alloc_trx.resize() +``` diff --git a/source/index.md b/source/index.md index 5912635..c3df194 100644 --- a/source/index.md +++ b/source/index.md @@ -24,8 +24,7 @@ growing computational needs of our field well into the future. ::::{grid} 2 :::{grid-item-card} TRX fundamentals ```{toctree} -:maxdepth: 1 -specification +:maxdepth: 2 concepts ``` ::: diff --git a/source/specification.md b/source/specification.md deleted file mode 100644 index 7a23b4e..0000000 --- a/source/specification.md +++ /dev/null @@ -1,175 +0,0 @@ -# TRX Specification - -This page describes the TRX file format specification. The authoritative specification is maintained in the [trx-spec](https://github.com/tee-ar-ex/trx-spec) repository. For more explanations -of the concepts underlying this specification, refer to {doc}`concepts`. -To learn more about the motivation for TRX, refer to {doc}`design`. - -## General description - -- Uncompressed ZIP file or simple folder architecture -- File architecture describes the data -- Each file basename is the metadata's name -- Each file extension is the metadata's dtype -- Each file dimension is in the value between basename and metadata -- 1-dimensional arrays do not have to follow this convention for readability -- All arrays have a C-style memory layout (row-major) -- All arrays have a little-endian byte order -- Compression is optional - - Use `ZIP_STORE` if no compression is desired - - Use `ZIP_DEFLATE` if compression is desired - - Compressed TRX files will have to be decompressed before being loaded - -## Header - -The header is primarily for human readability, read-time checks, and broader -compatibility. - -- Dictionary in JSON - - `VOXEL_TO_RASMM` — 4 lists of 4 floats (4x4 transformation matrix) - - `DIMENSIONS` — list of 3 uint16 - - `NB_STREAMLINES` — uint32 - - `NB_VERTICES` — uint64 - -## Arrays - -### positions - -- Written in world space (RASMM), like TCK files -- Always float16, float32, or float64 (default: float16) -- Stored as contiguous 3D array `(NB_VERTICES, 3)` - -### offsets - -- Always uint32 or uint64 -- Contains the index where each streamline starts in the positions array, - beginning at 0 -- Two ways of knowing how many vertices there are: - - Check the header - - Positions array size / dtype / 3 -- To get streamline lengths: append the total number of vertices to the end of - offsets and take the differences between consecutive elements - (`numpy.ediff1d`) - -### dpv (data per vertex) - -- Always of size `(NB_VERTICES, 1)` or `(NB_VERTICES, N)` - -### dps (data per streamline) - -- Always of size `(NB_STREAMLINES, 1)` or `(NB_STREAMLINES, N)` - -### groups - -Groups are tables of indices that allow sparse and overlapping representation -(clusters, connectomics, bundles). - -- All indices must satisfy `0 <= id < NB_STREAMLINES` -- Datatype should be uint32 -- Allow efficient extraction of a predefined streamline subset from memmaps -- Variable sizes across groups - -### dpg (data per group) - -- Each subfolder is named after its corresponding group -- Not all metadata need be present in all groups -- Always of size `(1,)` or `(N,)` - -## Accepted datatypes - -- int8, int16, int32, int64 -- uint8, uint16, uint32, uint64 -- float16, float32, float64 -- bit (for binary/boolean arrays) - -## Example structure - -``` -OHBM_demo.trx -├── dpg -│ ├── AF_L -│ │ ├── mean_fa.float16 -│ │ ├── shuffle_colors.3.uint8 -│ │ └── volume.uint32 -│ ├── AF_R -│ │ ├── mean_fa.float16 -│ │ ├── shuffle_colors.3.uint8 -│ │ └── volume.uint32 -│ ├── CC -│ │ ├── mean_fa.float16 -│ │ ├── shuffle_colors.3.uint8 -│ │ └── volume.uint32 -│ ├── CST_L -│ │ └── shuffle_colors.3.uint8 -│ ├── CST_R -│ │ └── shuffle_colors.3.uint8 -│ ├── SLF_L -│ │ ├── mean_fa.float16 -│ │ ├── shuffle_colors.3.uint8 -│ │ └── volume.uint32 -│ └── SLF_R -│ ├── mean_fa.float16 -│ ├── shuffle_colors.3.uint8 -│ └── volume.uint32 -├── dpv -│ ├── color_x.uint8 -│ ├── color_y.uint8 -│ ├── color_z.uint8 -│ └── fa.float16 -├── dps -│ ├── algo.uint8 -│ ├── algo.json -│ ├── clusters_QB.uint16 -│ ├── commit_colors.3.uint8 -│ └── commit_weights.float32 -├── groups -│ ├── AF_L.uint32 -│ ├── AF_R.uint32 -│ ├── CC.uint32 -│ ├── CST_L.uint32 -│ ├── CST_R.uint32 -│ ├── SLF_L.uint32 -│ └── SLF_R.uint32 -├── header.json -├── offsets.uint64 -└── positions.3.float16 -``` - -## Example code - -```python -from trx.trx_file_memmap import TrxFile, load, save -import numpy as np - -trx = load('complete_tractogram.trx') - -# Access the header (dict) / streamlines (ArraySequences) -trx.header -trx.streamlines - -# Access the dpv (dict) / dps (dict) -trx.data_per_vertex -trx.data_per_streamline - -# Access the groups (dict) / dpg (dict) -trx.groups -trx.data_per_group - -# Get a random subset of 10000 streamlines -indices = np.arange(len(trx.streamlines._lengths)) -np.random.shuffle(indices) -sub_trx = trx.select(indices[0:10000]) -save(sub_trx, 'random_1000.trx') - -# Get sub-groups only, from the random subset -for key in sub_trx.groups.keys(): - group_trx = sub_trx.get_group(key) - save(group_trx, '{}.trx'.format(key)) - -# Pre-allocate memmaps and append 100x the random subset -alloc_trx = TrxFile(nb_streamlines=1500000, nb_vertices=500000000, init_as=trx) -for i in range(100): - alloc_trx.append(sub_trx) - -# Resize to remove the unused portion of the memmap -alloc_trx.resize() -```