Skip to content

Selection functions from ingested SSI catalogues#465

Merged
joezuntz merged 43 commits into
masterfrom
ssi_selection_function
May 22, 2026
Merged

Selection functions from ingested SSI catalogues#465
joezuntz merged 43 commits into
masterfrom
ssi_selection_function

Conversation

@tmcornish
Copy link
Copy Markdown
Member

Added two stages to auxiliary_maps.py:

  • TXSelectionFunctionSSIMaps: Uses ingested SSI catalogues to estimate the survey selection function and its 1-sigma uncertainties in pixels containing injected sources. This makes use of a new function called make_dask_selection_function added to dr1.py.
  • TXModelSelectionFunction: Uses a model to extrapolate the selection function to the rest of the footprint, as defined via an input mask. For now this model is a multilinear fit to survey property maps, but there is scope for alternative models to be added later.

@tmcornish tmcornish self-assigned this Apr 28, 2026
@tmcornish
Copy link
Copy Markdown
Member Author

The current version is a basic working version and has been tested on DES Y3 SSI catalogues, but there are still a few TODOs to deal with before it's ready for merging:

  • The selection function in a given pixel is currently just estimated as the ratio of detected injections to the total number of injections in that pixel. Incorporating some means of applying lens sample-like cuts will give more useful estimates for clustering analyses.
  • Uncertainties on the selection function are approximated as Gaussian, but this breaks down if the number of detections in a pixel is either close to 0 or close to the total number of injections in that pixel.
  • Need to make TXModelSelectionFunction dask-compatible
  • Currently the best-fit parameters of the model and their covariance matrix are not saved anywhere.

@tmcornish
Copy link
Copy Markdown
Member Author

tmcornish commented Apr 30, 2026

Major changes since the last comment:

  • Removed TXSelectionFunctionSSIMaps since it was essentially doing a subset of what TXAuxiliarySSIMaps` already did.
  • Modified TXAuxiliaryMaps so that it also outputs the total injections per pixel as a function of magnitude threshold (this is so that it can be used in error estimation later on)
  • Modified TXModelSelectionFunction to instead use outputs from TXAuxiliaryMaps as inputs, namely the frac_det_by_mag_thresh and the (newly added) inj_count_by_mag_thresh maps. This stage now creates model predictions for each magnitude threshold.
  • TXModelSelectionFunction is now dask-compatible.
  • The best-fit parameters of the model and their covariance matrices are now saved to an HDF5 file.
  • Added config options for exclusion of pixels below a given coverage fraction and number of total injections.

I also found a bug where any attempt to load a mask with a non-zero threshold for removing pixels would produce an empty map, which should now be fixed.

@tmcornish tmcornish marked this pull request as ready for review April 30, 2026 18:25
Copy link
Copy Markdown
Collaborator

@elvinpoole elvinpoole left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great!

It might be useful to also upload an example ini file (or just add this stage to the existing SSI depth one)
Also, some of the methods (particularly polynomial_model) could do with a bigger docstring so we can see the inputs are dask arrays and what their expected shape will be
In general I wonder if the current dask logic is optimal. There are a few places where da.compute is called early so you lose the laziness of the dask arrays. This might be fine, just bringing it up in case there is a better way....

Comment thread txpipe/auxiliary_maps.py Outdated
Comment thread txpipe/auxiliary_maps.py
Comment thread txpipe/auxiliary_maps.py Outdated
Comment thread txpipe/auxiliary_maps.py
Comment thread txpipe/auxiliary_maps.py Outdated
Comment thread txpipe/auxiliary_maps.py Outdated
Comment thread txpipe/auxiliary_maps.py
Copy link
Copy Markdown
Collaborator

@joezuntz joezuntz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple of comments from me too.

Comment thread txpipe/auxiliary_maps.py Outdated
Comment thread txpipe/auxiliary_maps.py
Comment thread txpipe/auxiliary_maps.py
@tmcornish tmcornish requested review from elvinpoole and joezuntz May 11, 2026 15:16
@tmcornish tmcornish marked this pull request as draft May 11, 2026 17:30
@tmcornish tmcornish marked this pull request as draft May 11, 2026 17:30
@tmcornish tmcornish marked this pull request as ready for review May 11, 2026 18:55
Copy link
Copy Markdown
Collaborator

@elvinpoole elvinpoole left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks great! I'm happy with all the parts I previously commented on

@joezuntz joezuntz merged commit c8e9bc9 into master May 22, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants