Add container image documentation and generic Slurm/HTCondor submission scripts

changelog.md

@@ -2,6 +2,7 @@
 
 ### [Latest]
 
+- Add container image documentation and generic Slurm/HTCondor submission scripts with config-driven component enumeration [#160](https://github.com/umami-hep/umami-preprocessing/pull/160)
 ### [v0.3.2](https://github.com/umami-hep/umami-preprocessing/releases/tag/v0.3.2) (04.08.2026)
 
 - Add new configs for central dataset [#159](https://github.com/umami-hep/umami-preprocessing/pull/159)

docs/hpc.md

@@ -0,0 +1,207 @@
+# Running on HPC clusters
+
+The preprocessing stages can be parallelized over components, regions and splits using the
+`--component`, `--region` and `--split` flags described in [Run](run.md). On a Slurm or HTCondor
+cluster, each of these units of work can run as its own batch job inside the
+[container image](setup.md#container-image). UPP ships a small set of submission scripts in
+`scripts/batch/` that automate this.
+
+## Prerequisites
+
+- A cluster with Slurm or HTCondor and apptainer. On lxplus, HTCondor is provided by the CERN
+  batch service — see the [quickstart](https://batchdocs.web.cern.ch/local/quick.html) and
+  [job submission](https://batchdocs.web.cern.ch/local/submit.html) documentation.
+- A shared filesystem between the submitting node and the workers — the input, intermediate and
+  output files must be visible to all jobs. This is the case on lxplus (AFS/EOS) and typical
+  institute clusters.
+
+!!!info "Input/output data on lxplus (AFS/EOS)"
+
+    HTCondor on lxplus rejects submit files that reference EOS paths (executable, `output`,
+    `error`, `log`), so create the run directory — with the copied scripts and the `logs/`
+    directory — in your AFS work area and submit from there. The jobs themselves run with your
+    Kerberos credentials and can read and write `/afs` and `/eos` directly, so keep the large
+    input ntuples and outputs on EOS and bind both filesystems into the container:
+
+    ```bash
+    export UPP_BINDS=/afs,/eos,/tmp
+    ```
+
+    For very I/O-heavy workflows the batch service recommends staging data through the local
+    pool space of the job instead of writing to EOS directly — see
+    [Data flows](https://batchdocs.web.cern.ch/concepts/dataflow.html) and
+    [EOS](https://batchdocs.web.cern.ch/troubleshooting/eos.html) in the CERN batch docs.
+
+    If the run directory has to live on EOS, the experimental
+    [EosSubmit schedds](https://batchdocs.web.cern.ch/local/eossubmit.html)
+    (`module load lxbatch/eossubmit`) accept submit files with EOS paths, transferring all job
+    files via xrootd instead of using a shared filesystem. All submit file paths must then be on
+    EOS, and this mode has not been tested with these scripts.
+- The UPP container image (see [Container image](setup.md#container-image)). The scripts default to
+  the CVMFS-unpacked image
+  `/cvmfs/unpacked.cern.ch/gitlab-registry.cern.ch/aft/training-images/upp-images/upp:latest` when
+  it exists (no download or conversion needed) and fall back to
+  `docker://gitlab-registry.cern.ch/aft/training-images/upp-images/upp:latest` otherwise. Running
+  from `docker://` directly is fine: apptainer caches the converted image and only downloads again
+  when a new version is published. Alternatively, pin a specific local file with
+  `export UPP_IMAGE=/path/to/upp.sif` after an `apptainer pull`.
+
+!!!info "Apptainer cache location"
+
+    The apptainer cache defaults to `~/.apptainer/cache` and the conversion uses `/tmp` as
+    scratch space. On clusters with a small home quota (e.g. lxplus) point them to a larger
+    filesystem:
+
+    ```bash
+    export APPTAINER_CACHEDIR=/path/to/big/storage/apptainer_cache
+    export APPTAINER_TMPDIR=/path/to/big/storage/apptainer_tmp
+    ```
+
+## Interactive use
+
+For quick tests, run UPP inside the container on an interactive allocation:
+
+```bash
+salloc --ntasks 1 --cpus-per-task 4 --time 2:00:00
+srun apptainer exec --contain --pwd "$PWD" -B "$PWD" -B /home -B /tmp \
+    "$UPP_IMAGE" preprocess --config <path/to/config.yaml> --prep
+```
+
+## Batch submission scripts
+
+The `scripts/batch/` directory contains:
+
+- `submit.sh` runs on the login node. It reads the components from your preprocessing config and
+  submits one batch job per unit of work. The scheduler is auto-detected (`sbatch` found → Slurm,
+  `condor_submit` found → HTCondor) and can be forced with `--scheduler slurm|condor`.
+- `slurm_batch.sh` is the sbatch payload. It carries the `#SBATCH` resource header and starts the
+  container on the compute node.
+- `condor_job.sub` and `condor_batch.sh` are the HTCondor equivalents: the submit description with
+  the resource requests, and the job executable starting the container.
+- `run_stage.sh` runs inside the container and maps the submitted mode onto the `preprocess`
+  command line flags.
+
+To use them, create a run directory and copy the scripts. A clone of the repository is not
+required — the image contains the repository at `/workspace`, so the scripts can be taken straight
+from there:
+
+=== "CVMFS"
+
+    ```bash
+    mkdir my_preprocessing && cd my_preprocessing
+    cp -r /cvmfs/unpacked.cern.ch/gitlab-registry.cern.ch/aft/training-images/upp-images/upp:latest/workspace/scripts/batch .
+    ```
+
+=== "apptainer"
+
+    ```bash
+    mkdir my_preprocessing && cd my_preprocessing
+    apptainer exec docker://gitlab-registry.cern.ch/aft/training-images/upp-images/upp:latest \
+        cp -r /workspace/scripts/batch .
+    ```
+
+=== "local clone"
+
+    ```bash
+    mkdir my_preprocessing && cd my_preprocessing
+    cp -r <path/to/umami-preprocessing>/scripts/batch .
+    ```
+
+Adapt the resource specifications (number of CPUs, memory allocation, time limit, partition or
+account etc.) in the `slurm_batch.sh` (Slurm) or `condor_job.sub` (HTCondor) files to fit your
+needs. The `+JobFlavour` walltime flavours available on lxplus are listed in the
+[CERN batch documentation](https://batchdocs.web.cern.ch/local/submit.html).
+
+Then submit the stages in order, waiting for all jobs of one stage to finish before submitting the
+next:
+
+```bash
+./batch/submit.sh --config <path/to/config.yaml> --dry-run prepare  # preview only
+./batch/submit.sh --config <path/to/config.yaml> prepare
+./batch/submit.sh --config <path/to/config.yaml> resampling
+./batch/submit.sh --config <path/to/config.yaml> merge
+./batch/submit.sh --config <path/to/config.yaml> normalise
+./batch/submit.sh --config <path/to/config.yaml> plotting
+```
+
+Job logs are written to the `logs/output/` and `logs/error/` subdirectories of the current
+directory. Running `submit.sh` without a mode enters
+an interactive prompt for the mode and filters, and `./batch/submit.sh --help` prints all modes and
+options.
+
+On Slurm each job is submitted with its own `sbatch` call and job name. On HTCondor all jobs of one
+`submit.sh` invocation are submitted as a single cluster (one `condor_submit` with one process per
+job), with the job arguments written to `logs/condor_<mode>.args`.
+
+The available modes and the jobs they submit:
+
+| Mode | Jobs | `preprocess` flags per job |
+|------|------|----------------------------|
+| `sequential` | 1 | full chain (`--prep`, `--resample`, `--merge`, `--norm`, `--plot`) |
+| `prepare` | one per component and split | `--prep --component <c> --split <s>` |
+| `resampling` | one per region and split | `--resample --region <r> --split <s>` |
+| `fine_resampling` | one per component and split | `--resample --region <r> --component <c> --split <s>` |
+| `merge` | one per split | `--merge --split <s>` |
+| `normalise` | 1 | `--norm` |
+| `plotting` | one per split | `--plot --split <s>` |
+
+!!!warning "Stage ordering and parallel h5py access"
+
+    All jobs of a stage must finish before the next stage is submitted, e.g. all `prepare` jobs
+    before `resampling`. Also run the [initial sample check](run.md#additional-scripts-initial-sample-check)
+    once before submitting `prepare` jobs in parallel — it creates the virtual datasets which can
+    get corrupted when created by multiple jobs at once.
+
+## Config-driven job lists
+
+`submit.sh` never hardcodes which components exist. It calls the `list_components` script (part of
+UPP) to enumerate the components defined in the `components:` block of your config. No local UPP
+installation is needed for this: when `list_components` is not on the `PATH`, it is run inside the
+container image automatically. Only the `prepare`, `fine_resampling` and `resampling` modes (and
+the interactive mode) enumerate at all — the other modes submit without running UPP on the login
+node.
+
+```bash
+list_components --config <path/to/config.yaml>
+```
+
+```text
+lowpt   ttbar   bjets   lowpt_ttbar_bjets
+highpt  zprime  bjets   highpt_zprime_bjets
+...
+```
+
+Only combinations actually defined in the config are submitted. The selection can be narrowed with
+filter flags, each taking a comma- or space-separated list:
+
+```bash
+./batch/submit.sh --config <path/to/config.yaml> --regions lowpt --splits train prepare
+./batch/submit.sh --config <path/to/config.yaml> --samples ttbar --flavs "bjets,cjets" fine_resampling
+```
+
+Note that enumerating the components fully validates the config, so a broken config fails directly
+on the login node instead of inside the batch jobs.
+
+## Environment variables
+
+| Variable | Default | Purpose |
+|----------|---------|---------|
+| `UPP_IMAGE` | CVMFS-unpacked image if present, else `docker://...upp-images/upp:latest` | Container image (unpacked directory, local `.sif` or `docker://` URI) |
+| `UPP_BINDS` | `/home,/tmp` | Comma-separated paths bound into the container (the run and script directories are always bound in addition) |
+| `THROTTLE` | `30` | Seconds between `sbatch` calls (`0` disables; Slurm only) |
+| `DRY_RUN` | `0` | Set to `1` to print the submission commands instead of submitting |
+
+Make sure `UPP_BINDS` covers your input ntuples and output directory if they live outside `/home`
+(e.g. on a scratch filesystem), and export `UPP_IMAGE`/`UPP_BINDS` in your shell so they are also
+picked up by the batch jobs.
+
+!!!warning "Keep the throttle enabled"
+
+    The delay between `sbatch` calls avoids hammering the scheduler and gives jobs time to start
+    up without all of them hitting the shared filesystem at once. Only disable it for small
+    submissions.
+
+!!!info "Configs outside the repository"
+
+    When you copy a config out of the repository, `!include` directives with relative paths no
+    longer resolve. Use absolute paths in `!include` lines of copied configs.

docs/setup.md

@@ -128,6 +128,72 @@ python -m pip install .
     ```
 
 
+### Container image
+
+If you don't want to set up a Python environment at all, you can use the UPP container image.
+The CI builds `gitlab-registry.cern.ch/aft/training-images/upp-images/upp:latest` on every merge to
+`main` and a tagged image `upp:<tag>` (e.g. `upp:v0.3.1`) for every release. The image comes with UPP
+and its command line scripts (`preprocess`, `check_input_samples`, `list_components`) pre-installed.
+
+=== "apptainer"
+
+    On clusters (lxplus, HPC sites), apptainer can run the image directly from the registry:
+
+    ```bash
+    apptainer exec docker://gitlab-registry.cern.ch/aft/training-images/upp-images/upp:latest \
+        preprocess --config <path/to/config.yaml>
+    ```
+
+    The first `docker://` invocation converts the image to apptainer's SIF format, which takes a
+    while. The result is cached (default `~/.apptainer/cache`), so later invocations start quickly
+    and only download again when a new image version is published. If your home quota is small
+    (e.g. on lxplus), point the cache and the conversion scratch space to a larger filesystem:
+
+    ```bash
+    export APPTAINER_CACHEDIR=/path/to/big/storage/apptainer_cache
+    export APPTAINER_TMPDIR=/path/to/big/storage/apptainer_tmp
+    ```
+
+    On machines with CVMFS (lxplus, most HPC sites) the image is also distributed pre-unpacked
+    via [unpacked.cern.ch](https://gitlab.cern.ch/unpacked/sync), which skips the pull and
+    conversion entirely:
+
+    ```bash
+    apptainer exec /cvmfs/unpacked.cern.ch/gitlab-registry.cern.ch/aft/training-images/upp-images/upp:latest \
+        preprocess --config <path/to/config.yaml>
+    ```
+
+    The release tags are available there as well (e.g. `upp:v0.3.2`).
+
+    By default apptainer shares your home directory and working directory with the container. For a
+    cleaner environment use `--contain` and bind only the paths you need (your input ntuples and
+    output directory) with `-B`, keeping the working directory with `--pwd`:
+
+    ```bash
+    apptainer exec --contain --pwd "$PWD" -B "$PWD" -B /home -B /tmp -B <path/to/data> \
+        <image> preprocess --config <path/to/config.yaml>
+    ```
+
+    where `<image>` is any of the above (`docker://` URL, CVMFS path or local `.sif` file).
+
+=== "docker"
+
+    With docker, mount your working directory and data paths into the container:
+
+    ```bash
+    docker run --rm -it -v $PWD:$PWD -w $PWD \
+        gitlab-registry.cern.ch/aft/training-images/upp-images/upp:latest \
+        preprocess --config <path/to/config.yaml>
+    ```
+
+!!!info "Pin a release tag for production"
+
+    `upp:latest` follows the `main` branch and changes over time. For reproducible production
+    preprocessing, use a tagged release image like `upp:v0.3.1` instead.
+
+For running UPP as batch jobs on Slurm or HTCondor clusters with the container image, see
+[Running on HPC](hpc.md).
+
 ### Run the tests (Optional)
 
 To ensure that the package is working correctly, you can run the tests using the pytest framework. 

pyproject.toml

@@ -49,6 +49,7 @@ dev = [
 [project.scripts]
 preprocess = "upp.main:main"
 check_input_samples = "upp.utils.check_input_samples:main"
+list_components = "upp.utils.list_components:main"
 
 [tool.setuptools]
 packages = ["upp", "upp.classes", "upp.stages", "upp.utils"]

scripts/batch/README.md

@@ -0,0 +1,7 @@
+# Batch submission scripts
+
+Submit UPP preprocessing stages as Slurm or HTCondor batch jobs running inside the UPP container
+image.
+
+See the [Running on HPC](https://umami-hep.github.io/umami-preprocessing/hpc/) documentation page
+for usage instructions.

scripts/batch/condor_batch.sh

@@ -0,0 +1,15 @@
+#!/usr/bin/env bash
+# HTCondor job executable: starts the container on the worker node.
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+
+# Prefer the CVMFS-unpacked image when available, fall back to the registry
+DEFAULT_IMAGE="/cvmfs/unpacked.cern.ch/gitlab-registry.cern.ch/aft/training-images/upp-images/upp:latest"
+if [[ ! -e "${DEFAULT_IMAGE}" ]]; then
+    DEFAULT_IMAGE="docker://gitlab-registry.cern.ch/aft/training-images/upp-images/upp:latest"
+fi
+IMAGE="${UPP_IMAGE:-${DEFAULT_IMAGE}}"
+BINDS="${UPP_BINDS:-/home,/tmp}"
+
+apptainer exec --contain --pwd "${PWD}" -B "${BINDS}" -B "${PWD}" -B "${SCRIPT_DIR}" \
+    "${IMAGE}" "${SCRIPT_DIR}/run_stage.sh" "$@"

scripts/batch/condor_job.sub

@@ -0,0 +1,21 @@
+# HTCondor submit description used by submit.sh:
+#   condor_submit batch_dir=... batch_name=... upp_image=... upp_binds=... \
+#       condor_job.sub -queue "args from <args-file>"
+# Edit the resources below for your pool.
+request_cpus   = 4
+request_memory = 32 GB
+# CERN lxplus walltime flavour; ignored on other pools
+# Available flavours: https://batchdocs.web.cern.ch/local/submit.html
++JobFlavour    = "tomorrow"
+
+universe       = vanilla
+executable     = $(batch_dir)/condor_batch.sh
+arguments      = $(args)
+batch_name     = $(batch_name)
+environment    = "UPP_IMAGE=$(upp_image) UPP_BINDS=$(upp_binds)"
+output         = logs/output/$(ClusterId).$(ProcId).out
+error          = logs/error/$(ClusterId).$(ProcId).err
+log            = logs/$(ClusterId).log
+
+# Assumes a shared filesystem between submit and worker nodes
+should_transfer_files = NO