Add support for hdf5 files by elvinpoole · Pull Request #169 · LSSTDESC/healsparse

elvinpoole · 2026-01-20T20:21:10Z

Added read/write to a group in an hdf5 file

Useful for saving maps in TXPipe/ceci where we want N maps with different masks saved to the same file (e.g. galaxy density maps in different tomo bins)

Currently it only allows the whole map to the read (cannot specify list of coverage pixels with pixels) but this can be added

erykoff

I've started to go through but not the full way. The map should be saved and not the valid pixels. This is a key part of the healsparse format.

erykoff · 2026-01-20T21:09:04Z

        use_threads : `bool`, optional
           Use multithreaded reading for parquet files.
+        group : `str`, optional
+            if file is hdf5, read from this group


-> If file is hdf5, read from this group. (Please format as a sentence).
The next line is not necessary; the default is in the signature.

Actually, looking further, I think that this should be hdf_group instead of group everywhere.

erykoff · 2026-01-20T21:10:01Z

            coverage information.
+        group: `str`, optional
+            If file format is ``hdf5``, save to this group
+            otheriwse unused


-> If file format is ``hdf5``, save to this group; otherwise unused.
(Can fit on one line; next line is not necessary)

As above, make this hdf5_group.

erykoff · 2026-01-20T21:11:19Z

                         header=header, degrade_nside=degrade_nside,
-                         weightfile=weightfile, reduction=reduction, use_threads=use_threads)
+                         weightfile=weightfile, reduction=reduction, use_threads=use_threads,
+                         hdf5_group=group)


-> hdf_group=hdf5_group.

erykoff · 2026-01-20T21:11:36Z

        """
        _write_map(self, filename, clobber=clobber, nocompress=nocompress, format=format,
-                   nside_io=nside_io)
+                   nside_io=nside_io, hdf5_group=group)


-> hdf_group=hdf5_group.

erykoff · 2026-01-20T21:12:17Z

    use_threads : `bool`, optional
        Use multithreaded reading for parquet files.
+    hdf5_group: `str`, optional
+        hdf5 group to read from (only used if reading from an hdf5 file)


-> Read from this hdf5 group (only used reading from an hdf5 file).

erykoff · 2026-01-20T21:15:43Z

        raise IOError("Filename %s could not be found." % (filename))
    else:
-        raise NotImplementedError("HealSparse only supports fits and parquet files (with pyarrow).")
+        raise NotImplementedError("HealSparse only supports fits, hdf5 and parquet files (with pyarrow).")


-> ... hdf5 (with h5py), and parquet ...

erykoff · 2026-01-20T21:16:56Z

+    Parameters
+    ----------
+    hsp_map : HealSparseMap
+        Map to save


-> Map to save.

erykoff · 2026-01-20T21:17:20Z

+    filepath : str
+        HDF5 file path
+    group : str, optional
+        Name of the HDF5 group to store the map (default 'map')


No need to put the default in the docstring. Here and below.

erykoff · 2026-01-20T21:20:23Z

+        Overwrite the file/group if it exists (default False)
+    """
+    if os.path.isfile(filepath) and not clobber:
+        raise RuntimeError("Filename %s exists and clobber is False." % (filepath))


I'm not sure how this can work as is. clobber is the same as overwrite which isn't happening here. I think we can demand that the HDF5 already exists (document this!) and therefore (a) clobber does nothing and (b) we can demand that the file exists.

Or ... I see now that you're using clobber to overwrite an existing group? In which case, that needs to be documented but the check here is not what you want.

thanks. the clobber logic should now work. Yes, the intended behaviour is:
clobber=False only prevents overwriting the group. If the file already exists but the group is new, we can still save to a new group in that file (even if clobber=F)

erykoff · 2026-01-20T21:22:44Z

+
+        # Sparse map - only save valid pixels
+        valid_pixels = hsp_map.valid_pixels
+        grp.create_dataset("pixel", data=valid_pixels, compression="gzip")


This is not going to work, and in fact may explode well beyond your existing memory. The healsparse format is extremely efficient without recording pixel values (which can sometimes fail and is often slow). You want to save the map itself, and not valid pixels. Same thing with the coverage map.

This should now save the map

…r=False

erykoff

Thanks! This is looking a lot better but there are still a couple places that will not work in general (packed bool maps and recarrays). Also, please add to the documentation about this new format.

erykoff · 2026-01-22T21:02:30Z

            the ``healpix`` EXPLICIT format does not maintain all metadata and
            coverage information.
+        hdf5_group: `str`, optional
+            If file format is ``hdf5``, save to this group, otheriwse unused.


-> otherwise

erykoff · 2026-01-22T21:03:11Z

        is_fits_file = True
        fits.close()
-    except OSError:
+    except (OSError, UnicodeDecodeError):


Where is the UnicodeDecodeError coming from?

It seems this is what happens when you try to load the hdf5 file as if its fits. there might be a cleaner way of checking

Huh! Okay, could you add a comment here?

erykoff · 2026-01-22T21:08:45Z

+                )
+        elif hsp_map.is_bit_packed_map:
+            # save as bool array rather than packed to keep the same pixel as other maps when reading
+            sparse_map_reshape = np.asarray(hsp_map._sparse_map).reshape(ncov_in_sparse, nfine_per_cov)


This is going to blow up the memory and crash in the cases that a packed map is being used. Please use the packed format. In this case you just need to reshape with a divide-by-8 on nfine_per_cov e.g. https://github.com/LSSTDESC/healsparse/blob/main/healsparse/io_map_fits.py#L614-L616

erykoff · 2026-01-22T21:14:31Z

+
+            sparse_map = np.zeros(sparse_size, dtype=dtype)
+            for name, _ in dtype:
+                sparse_map[name][:] = sentinel


This isn't going to work because the sentinel only applies to primary. You'll need to read the overflow coverage pixel in all cases (not just here)

i read the overflow pixel from file anyway so this line was doing nothing. removed

erykoff · 2026-01-22T21:14:55Z

+            )
+        elif is_bit_packed:
+            sparse_map = grp["sparse_map"][cov_index_in_sparse_ordered, :][inv].reshape(-1)
+            sparse_map = _PackedBoolArray.from_boolean_array(sparse_map)


You'll need to leave this stored as packed.

erykoff

A few small comments. The only substantive one is that it looks like the "pixels read" tests are turned off, but it seems like these should work. So these tests should be turned on (and working) or specifying pixels needs to raise a NotImplementedError.

erykoff · 2026-01-27T21:29:47Z

+A :code:`HealSparseMap` may also be serialized to an HDF5 file. 
+Multiple :code:`HealSparseMap` objects can be stored in the same HDF5 file.
+Each :code:`HealSparseMap` is stored in a different HDF5 group.
+They are not required to have the same mask, :code:`nside_coverage` or :code:`nside_sparse`.


Oxford comma, please!

erykoff · 2026-01-28T16:29:53Z

        use_threads : `bool`, optional
           Use multithreaded reading for parquet files.
+        hdf5_group : `str`, optional
+           If file format is ``hdf5``, read from this group, otheriwse unused.


-> otherwise

erykoff · 2026-01-28T16:31:44Z

        is_fits_file = True
        fits.close()
-    except OSError:
+    except (OSError, UnicodeDecodeError):


Huh! Okay, could you add a comment here?

erykoff · 2026-01-28T16:33:27Z

+                    compression="gzip",
+                )
+        elif hsp_map.is_bit_packed_map:
+            # save as bool array rather than packed to keep the same pixel as other maps when reading


This comment is no longer operative.

erykoff · 2026-01-28T16:40:39Z

+import healsparse
+
+# read specific coverage pixels not yet implemented
+test_pixels_read = False


Is this still true?

I removed this as an option and made the pixel tests always run

elvinpoole added 15 commits January 19, 2026 17:20

added first pass at hdf5 read/write functions

a80ceb9

add missing underscore

b733a17

added function to check if hdf5 file

439e9c3

modified read/write methods to know about the hdf5 read/write

3510e9b

added pixels and degrade option to hdf5 read

54582b7

fixed test bugs in hdf5 function

959aedd

read hdf5 files to sparse_map instead of healpix map

2c638a2

added empty header return for hdf5

8de9fb4

removed sentinal handling

b03355a

added metadata header return

bf7065b

tidied comments

5365fde

assert that pixels is none for hdf5 read

0cea8f7

removed string from metadata saving in hdf5

5fb3431

added tests for hdf5 read/write

d8a4d97

formatting

7bfd256

elvinpoole marked this pull request as draft January 20, 2026 20:31

add h5py to install extras and actions

918ca8f

elvinpoole marked this pull request as ready for review January 20, 2026 20:43

erykoff requested changes Jan 20, 2026

View reviewed changes

elvinpoole added 11 commits January 21, 2026 20:02

save sparse map array as chunked dataset

a45d4d1

save coverage map index

498bd06

add out of order partial test

365a28f

fixed bug when doing partial read of out of order map

2ee09ba

added test for out of order partial read

576883e

black

4e5210c

fixed rec array bug

8c62ebe

added bit_packed read/write to hdf5

d4cb5fe

black

3f4d1de

black

09fd237

docstring fixes

b489258

elvinpoole added 3 commits January 22, 2026 15:01

added check if group exists inside hdf5 file when loading with clobbe…

c299b4d

…r=False

changed group->hdf5_group

28131d4

flake

9a8e575

erykoff requested changes Jan 22, 2026

View reviewed changes

elvinpoole added 6 commits January 23, 2026 12:04

fix typo in docstring

9f4d800

save bit packed map

d50efd2

removed sentinel fill in recarray read

62636b6

added tests for recarray and bit packed maps

d2e04f1

added hdf5 to documentation

9358cfe

formatting

f80221d

erykoff reviewed Jan 28, 2026

View reviewed changes

elvinpoole added 6 commits January 29, 2026 14:43

made test_pixels_read mandatory

5567fdd

oxford comma

e5b2478

fixed typo

8bb0e12

added comment on unicodeerror

6244cec

fixed otdated comments

5f79c45

fixed comment formatting

54023a4

erykoff changed the title ~~Hdf5~~ Add support for hdf5 files Jan 29, 2026

erykoff approved these changes Jan 29, 2026

View reviewed changes

erykoff merged commit ef36eb1 into LSSTDESC:main Jan 29, 2026
5 checks passed

Conversation

elvinpoole commented Jan 20, 2026

Uh oh!

erykoff left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

erykoff left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!