Generalize simulation production grid generation - less dependency on backend

[GernotMaier]

This pull request introduces a major refactor and improvement to the simulation production configuration workflow, focusing on a clear separation between observation grid generation and backend job submission. The changes streamline the process of generating executable simulation job grids, clarify command-line interfaces, and ensure that job submission tools operate on pre-generated job grids. The HTCondor job submission logic has been adapted to consume the new job grid format.

Simulation production workflow improvements:

• The production_generate_grid.py application now generates executable simulation job grids, supporting both explicit cartesian and axes-based configurations, and writes results as ECSV files.
• The HTCondor job submission tool (simulate_prod_htcondor_generator.py) now requires a pre-generated executable job grid file as input, removing the need for grid generation logic and related arguments.

API documentation updates:

• The API reference documentation has been reorganized to reflect the new modular structure, with clearer separation of modules such as corsika_limits_lookup, interpolation_handler, job_grid_io, observation_grid, and simulation_jobs.

HTCondor job submission code enhancements:

• The HTCondor script generator now reads job grids from the new job_grid_io module, and the parameter file writing logic has been updated to handle new fields (e.g., core_scatter_number, view_cone_min/max_value). The formatting and grouping of job specifications have been adjusted to match the new job grid structure. (src/simtools/job_execution/htcondor_script_generator.py)

The hope is that these changes result in a more maintainable, modular, and user-friendly simulation production pipeline.

[Copilot]

Pull request overview

This PR restructures simulation production preparation by decoupling grid/job definition from the execution backend (HTCondor). It introduces a serialized “job grid” (ECSV) as the contract between grid generation and submission, and refactors the existing grid/limits logic into dedicated modules.

Changes:

• Added new production-configuration modules for observation grid generation, CORSIKA limits interpolation, job-matrix expansion, and ECSV job-grid I/O.
• Updated the HTCondor generator to consume a pre-generated job grid (--job_grid_file) instead of deriving the grid internally.
• Reworked/relocated unit + integration tests and updated documentation/API references accordingly.

Reviewed changes

Copilot reviewed 36 out of 37 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tests/unit_tests/production_configuration/test_simulation_jobs.py	New unit coverage for simulation job expansion and clipping/scaling helpers.
tests/unit_tests/production_configuration/test_observation_grid.py	New unit coverage for ProductionGridEngine behavior (horizontal + ra_dec).
tests/unit_tests/production_configuration/test_job_grid_io.py	New unit coverage for job-grid ECSV serialization/deserialization.
tests/unit_tests/production_configuration/test_corsika_limits_lookup.py	New unit coverage for lookup-table filtering + interpolation.
tests/unit_tests/production_configuration/test_job_spec_builder.py	Removed tests for deleted job_spec_builder.
tests/unit_tests/production_configuration/test_generate_production_grid.py	Removed tests for deleted generate_production_grid (GridGeneration).
tests/unit_tests/layout/test_array_layout_utils.py	Moves by-version layout-name resolution tests onto array_layout_utils.resolve_array_layout_name.
tests/unit_tests/job_execution/test_htcondor_script_generator.py	Updates HTCondor generator tests to use job_grid_file and new params format.
tests/unit_tests/applications/test_simulate_prod_htcondor_generator.py	Updates CLI tests for new --job_grid_file argument and simplified build_application() usage.
tests/unit_tests/applications/test_production_generate_grid.py	New unit test ensuring production_generate_grid writes a job grid via serialize_job_grid.
tests/resources/production_job_grid_horizontal.ecsv	New sample job grid used by integration tests.
tests/resources/production_job_grid_horizontal_single.ecsv	New single-row sample job grid used by integration tests.
tests/resources/production_grid_points_horizontal.ecsv	Updates metadata coordinate_system to horizontal.
tests/resources/production_grid_generation_axes_definition_ra_dec.yml	New RA/Dec axes definition fixture.
tests/integration_tests/config/simulate_prod_htcondor_generator_grid_horizontal.yml	Integration config now passes job_grid_file instead of grid-definition parameters.
tests/integration_tests/config/simulate_prod_htcondor_generator_gamma_20_deg_north.yml	Same migration to job_grid_file for the single-case workflow.
tests/integration_tests/config/production_generate_grid.yml	Removed old integration workflow for the deleted grid generator.
tests/integration_tests/config/production_generate_grid_radec.yml	Removed old integration workflow for the deleted grid generator.
tests/integration_tests/config/production_generate_grid_ra_dec.yml	New integration config for RA/Dec job-grid generation (currently skipped due to DB requirement).
tests/integration_tests/config/production_generate_grid_horizontal.yml	New integration config for axes-based horizontal job-grid generation.
tests/integration_tests/config/production_generate_grid_horizontal_explicit.yml	New integration config for explicit-axes horizontal job-grid generation.
tests/integration_tests/config/production_derive_statistics.yml	Updates grid-points input file path to the new “horizontal” resource.
tests/integration_tests/config/plot_production_grid.yml	Updates grid-points input file path to the new “horizontal” resource.
src/simtools/utils/value_conversion.py	Adds get_value_in_unit() helper used by lookup interpolation.
src/simtools/utils/general.py	Extends ensure_list() to handle sets and JSON-like list strings.
src/simtools/production_configuration/simulation_jobs.py	New module: expands observation grids into full executable job rows (run expansion, clipping, scaling).
src/simtools/production_configuration/observation_grid.py	New ProductionGridEngine: generates observation grids and applies lookup limits; supports horizontal and ra_dec.
src/simtools/production_configuration/job_grid_io.py	New ECSV read/write for executable job grids with explicit schema.
src/simtools/production_configuration/corsika_limits_lookup.py	New lookup-table filtering + interpolation utility for CORSIKA limits.
src/simtools/production_configuration/job_spec_builder.py	Deleted (superseded by job-grid approach).
src/simtools/production_configuration/generate_production_grid.py	Deleted (superseded by ProductionGridEngine + job grid generation).
src/simtools/layout/array_layout_utils.py	Adds resolve_array_layout_name() to centralize by-version layout-name resolution.
src/simtools/job_execution/htcondor_script_generator.py	Updated to read a job grid (ECSV) and build HTCondor params/submit scripts from it.
src/simtools/applications/simulate_prod_htcondor_generator.py	CLI now requires --job_grid_file and no longer derives grids internally.
src/simtools/applications/production_generate_grid.py	App now generates and serializes executable job grids (ECSV) via build_simulation_jobs.
docs/source/api-reference/production_configuration.md	Updates API docs to reference new modules and removes deleted ones.
docs/changes/2190.feature.md	Adds changelog fragment for the production-grid/job-grid refactor.

[ctao-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
98.8% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube