PZ Server Training Set Maker

Creates customized training and validation/test sets using a compilation of spectroscopic redshifts and LSST photometric data.

Acknowledgements

Software developed and delivered as part of the in-kind contribution program BRA-LIN, from LIneA to the Rubin Observatory's LSST. An overview of this and other contributions is available here. The pipelines take advantage of the software support layer developed by LINCC, available as Python libraries: hats, hats-import and lsdb.

Tests

Test data

This repository currently contains a basic dataset, for testing purposes only. The ideal is to connect the pipelines to systems with access to a larger datasets.

Install

The only requirement is to have micromamba available in PATH:

git clone https://github.com/linea-it/pzserver_training_set_maker && cd pzserver_training_set_maker
./setup.sh
source env.sh

To install the pipeline at once:

./install.sh

The setup.sh will suggest a directory where the pipelines and datasets are installed, type 'yes' to confirm or 'no' to configure the desired path in each case with the respective environment variables and then run again setup.sh.

The installation script creates the pipe_tsm environment with micromamba.

By default the scripts use MAMBA_ROOT_PREFIX="$HOME/.micromamba". On a Slurm cluster, point this variable to a persistent location visible to the jobs if needed:

export MAMBA_ROOT_PREFIX=/path/to/shared/or/persistent/micromamba

Run a pipeline

To execute, simply:

# execute training set maker
mkdir process001
./run.sh config.yaml process001