v492 dev16

[GernotMaier]

This pull request primarily standardizes directory structures, improves configuration flexibility, and removes legacy support for "curved atmosphere" Monte Carlo (MC) data in various IRF (Instrument Response Function) production scripts. The changes also enhance the reproducibility and maintainability of training and job submission workflows by making parameters more configurable and by improving the handling of training and background files.

Key changes include:

Directory Structure Standardization:

• All scripts now consistently define log directories (LOGDIR) as subdirectories of their respective output directories (ODIR), rather than using longer, hardcoded paths. This improves clarity and maintainability across the codebase. [1] [2] [3] [4] [5]

Removal of Curved Atmosphere MC Handling:

• Scripts no longer preferentially look for or reference "curved atmosphere" MC input directories. All references and error messages have been updated to only consider "flat" (non-curved) MC data. This simplifies the workflow and reduces confusion. [1] [2] [3] [4]

Training and Parameterization Improvements:

• The XGB (Extreme Gradient Boosting) training script (IRF.trainXGBforGammaHadronSeparationTraining.sh) now reads zenith and noise bins directly from the JSON parameter file, rather than relying on hardcoded values. This makes training more flexible and easier to configure.
• The background file selection process for XGB training now samples files from all zenith bins, shuffles them, and limits the number per bin, ensuring better distribution and reproducibility.

Job Submission and Directory Handling Enhancements:

• Output directories are now created and permissions set earlier in the workflow, preventing potential errors and ensuring proper access for subsequent steps. [1] [2]
• The XGB training output directory in IRF.production.sh is now named with a consistent _XGB suffix, matching the updated workflow.

DispBDT Directory and Zenith Handling:

• The DispBDT directory logic in IRF.mscw_energy_MC_sub.sh is updated to use a _flat suffix and to select zenith angle subdirectories by degree (e.g., 20deg, 45deg) instead of legacy labels, with special handling for incomplete MC sets.

These updates collectively streamline the IRF production process, reduce legacy complexity, and make the scripts more robust and easier to maintain.

[Copilot]

Pull request overview

This PR is part of the v492 development cycle and refactors several IRF (Instrument Response Function) production shell scripts. It (a) disables/comments out the "curved atmosphere" MC simulation path while keeping it as a future re-enable marker (CURVED_ATMOSPHERE_MC), (b) consolidates the log directory to live under the output directory across multiple stages, (c) makes the XGB gamma/hadron training script read zenith and NSB binning from the JSON run-parameter file instead of hard-coded arrays, and (d) refines the zenith-angle-dependent disp BDT directory selection (replacing SZE/MZE/LZE/XZE buckets with explicit per-degree subdirectories and adding a flat-atmosphere subpath).

Changes:

• Disable curved-atmosphere input directories across mscw / lookup / TMVA / production scripts and update related log paths.
• Make XGB gamma/hadron training parameterize zenith bins and NSB levels from JSON, and randomize background file selection across all zenith bins via shuf.
• Refine disp BDT directory selection (_flat suffix, finer per-zenith bins, RedHV fallback handling) in IRF.mscw_energy_MC_sub.sh.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
scripts/IRF.trainXGBforGammaHadronSeparationTraining.sh	Read zenith/NSB bins from JSON; loop over NEZE zenith dirs; combine + reshuffle background lists via tmpfile and shuf.
scripts/IRF.trainTMVAforAngularReconstruction.sh	Move LOGDIR under ODIR; comment out curved INDIR; update error wording.
scripts/IRF.production.sh	Tweak test config comments; change BCKDIR from _DISP to _XGB; comment out curved simulation branch.
scripts/IRF.mscw_energy_MC.sh	Disable curved INDIR; place LOGDIR under ODIR; add ODIR mkdir/chmod.
scripts/IRF.generate_lookup_table_parts.sh	Disable curved INDIR; place LOGDIR under ODIR; update error wording.
scripts/IRF.generate_effective_area_parts.sh	Move LOGDIR under ODIR.
scripts/IRF.combine_lookup_table_parts.sh	Move LOGDIR under ODIR.
scripts/helper_scripts/IRF.mscw_energy_MC_sub.sh	Use ${ANATYPE}_flat (else branch only); replace SZE/MZE/LZE/XZE buckets with explicit per-degree dirs; add RedHV fallbacks at high zenith.
scripts/helper_scripts/IRF.dispXGB_sub.sh	Remove a stale comment.

Comments suppressed due to low confidence (1)

scripts/IRF.trainTMVAforAngularReconstruction.sh:101

• After this change, the fallback INDIR assignment inside if [[ ! -d $INDIR ]] is identical to the primary INDIR set above (both ${INDIRBASE}/ze${ZA}deg_offset${W}deg_NSB${N}MHz). The fallback block is now dead code — the directory check will never succeed on the second attempt if it failed the first. Either remove the redundant fallback, or restore a meaningful alternative path. The error message "(minus curved)" is also confusing now that no curved path is searched.

    if [[ ! -d $INDIR ]]; then
        INDIR="${INDIRBASE}/ze${ZA}deg_offset${W}deg_NSB${N}MHz"
        if [[ ! -d $INDIR ]]; then
            echo "Error, could not locate input directory. Locations searched (minus curved):"
            echo "$INDIR"
            exit 1
        fi
    fi

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.