diff --git a/backend/app/scripts/README.md b/backend/app/scripts/README.md index 711f360a..c9b6e09d 100644 --- a/backend/app/scripts/README.md +++ b/backend/app/scripts/README.md @@ -25,7 +25,9 @@ scripts/ │ ├── sites/ │ │ ├── lcrc-diagnostics-scanner.sh │ │ ├── nersc-diagnostics-scanner.sh -│ │ └── nersc.sh +│ │ ├── site_ingestion_launcher.sh +│ │ ├── chrysalis.config +│ │ └── nersc.config │ └── v3_data/ │ ├── __init__.py │ ├── lcrc-v3.env.example @@ -58,6 +60,7 @@ Example: python -m app.scripts.db.seed python -m app.scripts.db.rollback_seed python -m app.scripts.users.create_admin_account +python -m app.scripts.ingestion.hpc_upload_archive_ingestor python -m app.scripts.ingestion.nersc_archive_ingestor python -m app.scripts.ingestion.v3_data.lcrc_v3_archive_ingestor ``` @@ -120,6 +123,42 @@ If operational complexity increases, these scripts may later be consolidated int --- +## HPC Upload Archive Ingestor + +The scheduler-agnostic HPC upload archive ingestor is the preferred entrypoint for +site wrappers. It currently delegates to the existing NERSC archive ingestor, +preserving Perlmutter behavior while giving non-NERSC schedulers a stable shared +command. + +Example: + +```bash +uv run python -m app.scripts.ingestion.hpc_upload_archive_ingestor +``` + +### Site Collection Launcher + +`app/scripts/ingestion/sites/site_ingestion_launcher.sh` is the host-side +launcher for site collection. It loads `sites/.config`, then selects the +configured Python ingestor. Use it as: + +```bash +app/scripts/ingestion/sites/site_ingestion_launcher.sh nersc staging +app/scripts/ingestion/sites/site_ingestion_launcher.sh chrysalis archive +``` + +Each site config defines its machine name, archive roots, working and repository +paths, Python environment file, token export file, API base URL, archive lower +bound, and ingestor module. The launcher defaults to `DRY_RUN=true` with +`DRY_RUN_USE_REMOTE_STATE=true`, so it loads API credentials and performs +read-only state validation. Set `DRY_RUN_USE_REMOTE_STATE=false` for a +credential-free offline scan. Set `DRY_RUN=false` only after validating archive +access, token storage, network egress, and candidate counts. A capped +`MAX_CASES_PER_RUN` value limits real ingestion but still persists results. + +Site configs are operational inputs. Keep credentials in their referenced, +protected files rather than committing them to a config file. + ## NERSC Archive Ingestor The NERSC archive ingestor scans a bind-mounted performance archive directory, @@ -148,23 +187,19 @@ Configuration surface (via env vars): - `OLD_PERF_ARCHIVE_ROOT` (default `/OLD_PERF` for `SCAN_MODE=archive`) - `MACHINE_NAME` (default `perlmutter`) - `DRY_RUN` (default `true`) +- `DRY_RUN_USE_REMOTE_STATE` (default `true`; set `false` for offline dry runs) - `MAX_CASES_PER_RUN` (optional, default not set) - `MAX_ATTEMPTS` (optional, default not set) - `REQUEST_TIMEOUT_SECONDS` (optional, default 60) - `ARCHIVE_YEAR_START` (optional, archive mode only; accepts `YYYY` or `YYYY-MM`) - `ARCHIVE_YEAR_END` (optional, archive mode only; accepts `YYYY` or `YYYY-MM`) -Helper wrapper: - -- `backend/app/scripts/ingestion/sites/nersc.sh` activates `backend/.venv`, sets the documented NERSC staging and archive roots, defaults to `SCAN_MODE=archive`, defaults to `DRY_RUN=true`, and then runs `python -m app.scripts.ingestion.nersc_archive_ingestor`. -- Override `SCAN_MODE`, `DRY_RUN`, or any other supported env var in the caller or cron entry when you need a different schedule or behavior. - Archive notes: - Archive mode traverses only top-level `YYYY-MM` directories under `OLD_PERF_ARCHIVE_ROOT`. Other top-level directories are ignored. - Archive scans may include paths without a `COMPLETED/` directory. When snapshot status buckets exist, ingestor scans only `COMPLETED/` and ignores sibling directories in that snapshot bucket. - Archive dedupe is based on logical case identity plus `execution_id`, not the full timestamped snapshot path. -- `ARCHIVE_YEAR_START` / `ARCHIVE_YEAR_END` are intended for scoped backfills so operators can avoid scanning the full historical tree when unnecessary. +- Direct Python entrypoints leave `ARCHIVE_YEAR_START` / `ARCHIVE_YEAR_END` unset. The site collection launcher applies each site's configured archive lower bound; callers may override either bound for a differently scoped archive scan. - `YYYY` values expand to full-year bounds (`START=2020` means `2020-01`; `END=2020` means `2020-12`), while `YYYY-MM` values target exact archive month buckets. ## One-Time Chrysalis E3SM v3 Archive Backfill @@ -288,6 +323,7 @@ Configuration surface (via env vars): - `OLD_PERF_ARCHIVE_ROOT` (default `/OLD_PERF` for `SCAN_MODE=archive`) - `MACHINE_NAME` (default `perlmutter`) - `DRY_RUN` (default `true`) +- `DRY_RUN_USE_REMOTE_STATE` (default `true`; set `false` for offline dry runs) - `MAX_CASES_PER_RUN` (optional, default not set) - `MAX_ATTEMPTS` (optional, default not set) - `REQUEST_TIMEOUT_SECONDS` (optional, default 60) diff --git a/backend/app/scripts/ingestion/archive_ingestor_core.py b/backend/app/scripts/ingestion/archive_ingestor_core.py index 7fa53416..64fa2ad8 100644 --- a/backend/app/scripts/ingestion/archive_ingestor_core.py +++ b/backend/app/scripts/ingestion/archive_ingestor_core.py @@ -132,6 +132,7 @@ "startup_configuration_runtime": ( "machine_name", "dry_run", + "dry_run_use_remote_state", "max_cases_per_run", "max_attempts", "request_timeout_seconds", @@ -261,6 +262,8 @@ class IngestorConfig: archive_year_start: str | None = None # Optional archive upper bound normalized to a YYYY-MM archive bucket. archive_year_end: str | None = None + # Whether a dry run reads existing state and archive checkpoints from SimBoard. + dry_run_use_remote_state: bool = True class IngestionRequestError(Exception): @@ -504,6 +507,9 @@ def _build_config_from_env( machine_name = os.getenv("MACHINE_NAME", DEFAULT_MACHINE_NAME) dry_run = _parse_bool(os.getenv("DRY_RUN"), default=True) + dry_run_use_remote_state = _parse_bool( + os.getenv("DRY_RUN_USE_REMOTE_STATE"), default=True + ) max_cases_per_run = _parse_optional_int(os.getenv("MAX_CASES_PER_RUN")) if max_cases_per_run is not None and max_cases_per_run <= 0: @@ -557,6 +563,7 @@ def _build_config_from_env( machine_name=machine_name, scan_mode=cast(Literal["staging", "archive"], scan_mode), dry_run=dry_run, + dry_run_use_remote_state=dry_run_use_remote_state, max_cases_per_run=max_cases_per_run, max_attempts=max_attempts, request_timeout_seconds=timeout_seconds, diff --git a/backend/app/scripts/ingestion/archive_workflow.py b/backend/app/scripts/ingestion/archive_workflow.py index cdaf54e5..985754fc 100644 --- a/backend/app/scripts/ingestion/archive_workflow.py +++ b/backend/app/scripts/ingestion/archive_workflow.py @@ -48,7 +48,7 @@ def _validate_run_preconditions( ) return False - if not config.api_token: + if (not config.dry_run or config.dry_run_use_remote_state) and not config.api_token: log_event_fn( "configuration_error", {"error": "SIMBOARD_API_TOKEN is required"}, @@ -90,6 +90,7 @@ def _log_startup_configuration( { "machine_name": config.machine_name, "dry_run": config.dry_run, + "dry_run_use_remote_state": config.dry_run_use_remote_state, "max_cases_per_run": config.max_cases_per_run, "max_attempts": config.max_attempts, "request_timeout_seconds": config.request_timeout_seconds, diff --git a/backend/app/scripts/ingestion/hpc_upload_archive_ingestor.py b/backend/app/scripts/ingestion/hpc_upload_archive_ingestor.py index 73c9a475..a38bac72 100644 --- a/backend/app/scripts/ingestion/hpc_upload_archive_ingestor.py +++ b/backend/app/scripts/ingestion/hpc_upload_archive_ingestor.py @@ -5,7 +5,7 @@ is read from environment variables (for example ``SIMBOARD_API_BASE_URL``, ``SIMBOARD_API_TOKEN``, ``PERF_ARCHIVE_ROOT``, ``OLD_PERF_ARCHIVE_ROOT``, and ``DRY_RUN``). -Each ingest run executes these phases: +Non-dry-run ingestion executes these phases: 1. In archive mode, fetch completed snapshot checkpoints. 2. Fetch persisted per-case state from SimBoard API. @@ -13,8 +13,10 @@ 4. Persist discovery results, then package and submit each changed case. 5. In archive mode, settle and persist completed snapshot checkpoints. -Dry runs stop after discovery and emit a summary. Successful ingestions update -database state used to keep future runs idempotent. +Dry runs read remote state and checkpoints by default, stop after discovery, +and emit a summary without writes. Set ``DRY_RUN_USE_REMOTE_STATE=false`` for +an offline dry run with empty local state. Successful ingestions update database +state used to keep future runs idempotent. Structured log metric definitions for this runner live in ``docs/architecture/metadata-ingestion.md``. This module emits those field names @@ -33,7 +35,7 @@ from collections.abc import Iterator from contextlib import contextmanager from pathlib import Path -from typing import Callable +from typing import Any, Callable from app.features.ingestion.parsers.parser import _locate_metadata_files from app.scripts.ingestion.archive_client import ( @@ -63,6 +65,7 @@ MetadataLocator, SleepCallback, _build_config_from_env, + _fresh_state, _log_event, ) from app.scripts.ingestion.archive_workflow import ( @@ -117,21 +120,13 @@ def _case_submission_callback( ) -def _run_ingestor( +def _prepare_run_state( config: IngestorConfig, - metadata_locator: MetadataLocator = _locate_metadata_files, - sleep_fn: SleepCallback = time.sleep, - post_request_fn: CaseSubmissionCallback | None = None, - discovery_post_request_fn: DiscoveryResultsPersistenceCallback | None = None, - checkpoint_post_request_fn: ArchiveCheckpointPersistenceCallback | None = None, - case_path_filter: Callable[[Path], bool] | None = None, - additional_dir_pruner: Callable[[str, list[str]], None] | None = None, archive_checkpointing: bool = True, - run_report: IngestorRunReport | None = None, -) -> int: - """Execute one complete archive scan-and-upload cycle.""" - use_prepared_archives = post_request_fn is None - post_request_fn = _case_submission_callback(post_request_fn) +) -> tuple[dict[str, Any], set[str], str] | None: + """Build offline dry-run state or fetch state needed for this run.""" + if config.dry_run and not config.dry_run_use_remote_state: + return _fresh_state(), set(), "" endpoint_url = _build_endpoint_url(config) state_endpoint_url = _build_state_endpoint_url(config) @@ -141,10 +136,6 @@ def _run_ingestor( state_endpoint_url=state_endpoint_url, log_event_fn=_log_event, ) - - if not _validate_run_preconditions(config, log_event_fn=_log_event): - return 1 - completed_snapshot_keys: set[str] = set() if config.scan_mode == "archive" and archive_checkpointing: try: @@ -162,7 +153,7 @@ def _run_ingestor( "archive_checkpoint_fetch_failed", {"status_code": exc.status_code, "error": str(exc)}, ) - return 1 + return None try: state = _fetch_ingestion_state( @@ -180,7 +171,37 @@ def _run_ingestor( "error": str(exc), }, ) + return None + + return state, completed_snapshot_keys, endpoint_url + + +def _run_ingestor( + config: IngestorConfig, + metadata_locator: MetadataLocator = _locate_metadata_files, + sleep_fn: SleepCallback = time.sleep, + post_request_fn: CaseSubmissionCallback | None = None, + discovery_post_request_fn: DiscoveryResultsPersistenceCallback | None = None, + checkpoint_post_request_fn: ArchiveCheckpointPersistenceCallback | None = None, + case_path_filter: Callable[[Path], bool] | None = None, + additional_dir_pruner: Callable[[str, list[str]], None] | None = None, + archive_checkpointing: bool = True, + run_report: IngestorRunReport | None = None, +) -> int: + """Execute one complete archive scan-and-upload cycle.""" + use_prepared_archives = post_request_fn is None + post_request_fn = _case_submission_callback(post_request_fn) + + if not _validate_run_preconditions(config, log_event_fn=_log_event): + return 1 + + run_state = _prepare_run_state( + config, + archive_checkpointing=archive_checkpointing, + ) + if run_state is None: return 1 + state, completed_snapshot_keys, endpoint_url = run_state new_discovery_results: list[ExecutionDiscoveryResult] = [] try: diff --git a/backend/app/scripts/ingestion/nersc_archive_ingestor.py b/backend/app/scripts/ingestion/nersc_archive_ingestor.py index 288655a1..e3546806 100644 --- a/backend/app/scripts/ingestion/nersc_archive_ingestor.py +++ b/backend/app/scripts/ingestion/nersc_archive_ingestor.py @@ -5,7 +5,7 @@ from environment variables (for example ``SIMBOARD_API_BASE_URL``, ``SIMBOARD_API_TOKEN``, ``PERF_ARCHIVE_ROOT``, ``OLD_PERF_ARCHIVE_ROOT``, and ``DRY_RUN``). -Each ingest run executes these phases: +Non-dry-run ingestion executes these phases: 1. Fetch persisted per-case state from SimBoard API. 2. In archive mode, fetch completed snapshot checkpoints. @@ -13,8 +13,10 @@ 4. Persist discovery results, then submit each changed case with retry/backoff. 5. In archive mode, settle and persist completed snapshot checkpoints. -Dry runs stop after discovery and emit a summary. Successful ingestions update -database state used to keep future runs idempotent. +Dry runs read remote state and checkpoints by default, stop after discovery, +and emit a summary without writes. Set ``DRY_RUN_USE_REMOTE_STATE=false`` for +an offline dry run with empty local state. Successful ingestions update database +state used to keep future runs idempotent. Structured log metric definitions for this runner live in ``docs/architecture/metadata-ingestion.md``. This module emits those field names @@ -24,6 +26,7 @@ from __future__ import annotations import time +from typing import Any from app.features.ingestion.parsers.parser import _locate_metadata_files from app.scripts.ingestion.archive_client import ( @@ -47,6 +50,7 @@ SleepCallback, UnsupportedArchiveLayoutError, _build_config_from_env, + _fresh_state, _log_event, ) from app.scripts.ingestion.archive_workflow import ( @@ -104,31 +108,12 @@ def _case_submission_callback( return _post_ingestion_request if post_request_fn is None else post_request_fn -def _run_ingestor( +def _prepare_run_state( config: IngestorConfig, - metadata_locator: MetadataLocator = _locate_metadata_files, - sleep_fn: SleepCallback = time.sleep, - post_request_fn: CaseSubmissionCallback | None = None, - discovery_post_request_fn: DiscoveryResultsPersistenceCallback | None = None, - checkpoint_post_request_fn: ArchiveCheckpointPersistenceCallback | None = None, -) -> int: - """Execute one complete archive scan-and-ingest cycle. - - Parameters - ---------- - config : IngestorConfig - Runtime configuration values. - metadata_locator : Callable[[str], object], optional - Validation callable used when scanning execution directories. - sleep_fn : Callable[[float], None], optional - Sleep function used for retry backoff. - - Returns - ------- - int - Process exit code (``0`` success, ``1`` failure). - """ - post_request_fn = _case_submission_callback(post_request_fn) +) -> tuple[dict[str, Any], set[str], str] | None: + """Build offline dry-run state or fetch state needed for this run.""" + if config.dry_run and not config.dry_run_use_remote_state: + return _fresh_state(), set(), "" endpoint_url = _build_endpoint_url(config) state_endpoint_url = _build_state_endpoint_url(config) @@ -138,10 +123,6 @@ def _run_ingestor( state_endpoint_url=state_endpoint_url, log_event_fn=_log_event, ) - - if not _validate_run_preconditions(config, log_event_fn=_log_event): - return 1 - try: state = _fetch_ingestion_state( state_endpoint_url, @@ -158,7 +139,7 @@ def _run_ingestor( "error": str(exc), }, ) - return 1 + return None completed_snapshot_keys: set[str] = set() if config.scan_mode == "archive": @@ -177,7 +158,44 @@ def _run_ingestor( "archive_checkpoint_fetch_failed", {"status_code": exc.status_code, "error": str(exc)}, ) - return 1 + return None + + return state, completed_snapshot_keys, endpoint_url + + +def _run_ingestor( + config: IngestorConfig, + metadata_locator: MetadataLocator = _locate_metadata_files, + sleep_fn: SleepCallback = time.sleep, + post_request_fn: CaseSubmissionCallback | None = None, + discovery_post_request_fn: DiscoveryResultsPersistenceCallback | None = None, + checkpoint_post_request_fn: ArchiveCheckpointPersistenceCallback | None = None, +) -> int: + """Execute one complete archive scan-and-ingest cycle. + + Parameters + ---------- + config : IngestorConfig + Runtime configuration values. + metadata_locator : Callable[[str], object], optional + Validation callable used when scanning execution directories. + sleep_fn : Callable[[float], None], optional + Sleep function used for retry backoff. + + Returns + ------- + int + Process exit code (``0`` success, ``1`` failure). + """ + post_request_fn = _case_submission_callback(post_request_fn) + + if not _validate_run_preconditions(config, log_event_fn=_log_event): + return 1 + + run_state = _prepare_run_state(config) + if run_state is None: + return 1 + state, completed_snapshot_keys, endpoint_url = run_state new_discovery_results: list[ExecutionDiscoveryResult] = [] try: diff --git a/backend/app/scripts/ingestion/sites/Setup_for_Collection_Scripts_and_Crontab.md b/backend/app/scripts/ingestion/sites/Setup_for_Collection_Scripts_and_Crontab.md new file mode 100644 index 00000000..eaa59846 --- /dev/null +++ b/backend/app/scripts/ingestion/sites/Setup_for_Collection_Scripts_and_Crontab.md @@ -0,0 +1,92 @@ +# Setup for Collection Scripts and Crontab + +## Enables + +- standardized routine remote site metadata collection for Simboard backend ingestion + +## Important Locations + +```text +[SIMBOARD_ROOT]/ User/Operator-selected read/write/execute directory +[SIMBOARD_ROOT]/repository/simboard Location of git-cloned "simboard" repository +[SIMBOARD_ROOT]/operations Initial work directory for crontab execution, overflow logs, notes +``` + +## Important Files + +```text +[SIMBOARD_ROOT]/repository/simboard/backend/app/scripts/ingestion/sites/.config + + These are the site-specific environment variables the site-launch script will export. + +[SIMBOARD_ROOT]/operations/.api_token_export + + Holds the export command that will set the SIMBOARD_API_TOKEN variable when sourced in + the cron'd site_ingestion_launcher.sh script. This adds a layer of indirection to prevent + accidental storage of the token in the public repository. +``` + +## Procedures + +Assuming the "SIMBOARD_ROOT", repository and operations directories are set, and the +site configuration file is properly defined in + + [SIMBOARD_ROOT]/repository/simboard/backend/app/scripts/ingestion/sites/.config + +the following command line, established in your crontab file, will serve to run the background +collection in "staging" or "archive" modes, respectively: + +```text + export SIMBOARD_ROOT=/lcrc/group/e3sm2/simboard \ + && cd ${SIMBOARD_ROOT}/operations \ + && ${SIMBOARD_ROOT}/repository/simboard/backend/app/scripts/ingestion/sites/site_ingestion_launcher.sh chrysalis staging + + export SIMBOARD_ROOT=/lcrc/group/e3sm2/simboard \ + && cd ${SIMBOARD_ROOT}/operations \ + && ${SIMBOARD_ROOT}/repository/simboard/backend/app/scripts/ingestion/sites/site_ingestion_launcher.sh chrysalis archive +``` + +These command should be preceded by crontab scedule specifications, recommended to be: + +```text + 5 20 35 50 * * * * (for the "staging" operation, every 15 minutes of every hour) + 10 15 * * * (for the "archive" operaiton, 15:10 every day) +``` + + +## About the Site Config + +The following sample (from Chrysalis) demonstrates certain flexibilities involved: + +```text + # export SIMBOARD_WORKDIR="/lcrc/group/e3sm2/simboard/operations" + # export SIMBOARD_MODULES="/lcrc/group/e3sm2/simboard/repository/simboard/backend" + export SIMBOARD_ENV_FILE="${HOME}/envs/test_simboard/bin/activate" + export SIMBOARD_API_TOKEN_FILE="${SIMBOARD_WORKDIR}/.api_token_export" + export SIMBOARD_INGESTOR_MODULE="app.scripts.ingestion.hpc_upload_archive_ingestor" + export SIMBOARD_API_BASE_URL="${SIMBOARD_API_BASE_URL:-https://simboard-dev-api.e3sm.org}" + export SIMBOARD_DEFAULT_ARCHIVE_YEAR_START="${SIMBOARD_DEFAULT_ARCHIVE_YEAR_START:-2025-01}" + export DRY_RUN="${DRY_RUN:-true}" + export PERF_ARCHIVE_ROOT="${PERF_ARCHIVE_ROOT:-/lcrc/group/e3sm/PERF_Chrysalis/performance_archive}" + export OLD_PERF_ARCHIVE_ROOT="${OLD_PERF_ARCHIVE_ROOT:-/lcrc/group/e3sm/PERF_Chrysalis/OLD_PERF}" + export MACHINE_NAME="${MACHINE_NAME:-chrysalis}" +``` + +Note that the first two variables (SIMBOARD_WORKDIR, SIMBOARD_MODULES) can be set automatically in +the launcher script, defined by their relation to SIMBOARD_ROOT. Likewise, we could eliminate +SIMBOARD_INGESTOR_MODULE, as this fixed string is not site-dependent and could be defined in the +laucher script. The same is true for SIMBOARD_API_BASE_URL, and although MACHINE_NAME is site +specific, it is supplied on the crontab command line and could be exported in the launch script. + +Hence, the minimal site configuration script might look like: + +```text + export SIMBOARD_ENV_FILE="${HOME}/envs/test_simboard/bin/activate" + export SIMBOARD_API_TOKEN_FILE="${SIMBOARD_WORKDIR}/.api_token_export" + export SIMBOARD_API_BASE_URL="${SIMBOARD_API_BASE_URL:-https://simboard-dev-api.e3sm.org}" + export SIMBOARD_DEFAULT_ARCHIVE_YEAR_START="${SIMBOARD_DEFAULT_ARCHIVE_YEAR_START:-2025-01}" + export DRY_RUN="${DRY_RUN:-true}" + export PERF_ARCHIVE_ROOT="${PERF_ARCHIVE_ROOT:-/lcrc/group/e3sm/PERF_Chrysalis/performance_archive}" + export OLD_PERF_ARCHIVE_ROOT="${OLD_PERF_ARCHIVE_ROOT:-/lcrc/group/e3sm/PERF_Chrysalis/OLD_PERF}" +``` + diff --git a/backend/app/scripts/ingestion/sites/chrysalis.config b/backend/app/scripts/ingestion/sites/chrysalis.config new file mode 100644 index 00000000..5548bf2b --- /dev/null +++ b/backend/app/scripts/ingestion/sites/chrysalis.config @@ -0,0 +1,11 @@ +export SIMBOARD_WORKDIR="/lcrc/group/e3sm2/simboard/operations" +export SIMBOARD_MODULES="/lcrc/group/e3sm2/simboard/repository/simboard/backend" +export SIMBOARD_ENV_FILE="${HOME}/envs/test_simboard/bin/activate" +export SIMBOARD_API_TOKEN_FILE="${SIMBOARD_WORKDIR}/.api_token_export" +export SIMBOARD_INGESTOR_MODULE="app.scripts.ingestion.hpc_upload_archive_ingestor" +export SIMBOARD_API_BASE_URL="${SIMBOARD_API_BASE_URL:-https://simboard-dev-api.e3sm.org}" +export SIMBOARD_DEFAULT_ARCHIVE_YEAR_START="${SIMBOARD_DEFAULT_ARCHIVE_YEAR_START:-2025-01}" +export DRY_RUN="${DRY_RUN:-true}" +export PERF_ARCHIVE_ROOT="${PERF_ARCHIVE_ROOT:-/lcrc/group/e3sm/PERF_Chrysalis/performance_archive}" +export OLD_PERF_ARCHIVE_ROOT="${OLD_PERF_ARCHIVE_ROOT:-/lcrc/group/e3sm/PERF_Chrysalis/OLD_PERF}" +export MACHINE_NAME="${MACHINE_NAME:-chrysalis}" diff --git a/backend/app/scripts/ingestion/sites/crontab.example b/backend/app/scripts/ingestion/sites/crontab.example new file mode 100644 index 00000000..037a5e16 --- /dev/null +++ b/backend/app/scripts/ingestion/sites/crontab.example @@ -0,0 +1,27 @@ +# Example crontab entries for a generic site collection launcher. +# Install with something like: +# crontab backend/app/scripts/ingestion/sites/.crontab.example + +SHELL=/bin/bash +PATH=/usr/local/bin:/usr/bin:/bin +CRON_TZ=UTC + +# Assume SIMBOARD_ROOT is a directory containing both +# "repository/simboard/backend/..." and +# "operations" subdirectories. +export SIMBOARD_ROOT= + +# Staging scan: run every 15 minutes, offset 5 minutes into the hour. +5 20 35 50 * * * * cd ${SIMBOARD_ROOT}/operations && ${SIMBOARD_ROOT}/repository/simboard/backend/app/scripts/ingestion/sites/site_ingestion_launcher.sh staging + +# Archive scan: run daily at 03:15 UTC. The site config supplies the default +# lower bound. Override `ARCHIVE_YEAR_START` or `ARCHIVE_YEAR_END` in the cron +# environment for a differently scoped scan. Values may use YYYY or YYYY-MM. +10 15 * * * cd ${SIMBOARD_ROOT}/operations && ${SIMBOARD_ROOT}/repository/simboard/backend/app/scripts/ingestion/sites/site_ingestion_launcher.sh chrysalis archive + +# I DON'T KNOW ABOUT THESE +# Diagnostics provenance scan: start dry-run, inspect logs, then set DRY_RUN=false. +# 20 * * * * cd ${REPO_DIR} && ${REPO_DIR}/backend/app/scripts/ingestion/sites/nersc-diagnostics-scanner.sh >> ${REPO_DIR}/backend/app/scripts/ingestion/sites/nersc-diagnostics-scanner.log 2>&1 + +# Chrysalis diagnostics provenance scan; use its local checkout path for REPO_DIR. +# 25 * * * * cd ${REPO_DIR} && MACHINE_NAME=chrysalis ${REPO_DIR}/backend/app/scripts/ingestion/sites/lcrc-diagnostics-scanner.sh >> ${REPO_DIR}/backend/app/scripts/ingestion/sites/lcrc-diagnostics-scanner.log 2>&1 diff --git a/backend/app/scripts/ingestion/sites/nersc.config b/backend/app/scripts/ingestion/sites/nersc.config new file mode 100644 index 00000000..f220a3f0 --- /dev/null +++ b/backend/app/scripts/ingestion/sites/nersc.config @@ -0,0 +1,12 @@ +export SIMBOARD_WORKDIR="${HOME}/Ops/simboard" +export SIMBOARD_REPODIR="${HOME}/gitrepo/simboard/backend" +export SIMBOARD_ENV_FILE="${HOME}/envs/test_simboard/bin/activate" +export SIMBOARD_API_TOKEN_FILE="${SIMBOARD_WORKDIR}/.api_token_export" +export SIMBOARD_INGESTOR_MODULE="app.scripts.ingestion.nersc_archive_ingestor" +export SIMBOARD_API_BASE_URL="${SIMBOARD_API_BASE_URL:-https://simboard-dev-api.e3sm.org}" +export SIMBOARD_DEFAULT_ARCHIVE_YEAR_START="${SIMBOARD_DEFAULT_ARCHIVE_YEAR_START:-2025-01}" + +export DRY_RUN="${DRY_RUN:-true}" +export PERF_ARCHIVE_ROOT="${PERF_ARCHIVE_ROOT:-/global/cfs/projectdirs/e3sm/performance_archive}" +export OLD_PERF_ARCHIVE_ROOT="${OLD_PERF_ARCHIVE_ROOT:-/global/cfs/projectdirs/e3sm/OLD_PERF}" +export MACHINE_NAME="${MACHINE_NAME:-perlmutter}" diff --git a/backend/app/scripts/ingestion/sites/nersc.crontab.example b/backend/app/scripts/ingestion/sites/nersc.crontab.example deleted file mode 100644 index 4ec33111..00000000 --- a/backend/app/scripts/ingestion/sites/nersc.crontab.example +++ /dev/null @@ -1,32 +0,0 @@ -# Example crontab entries for the NERSC ingestion wrapper. -# Install with something like: -# crontab backend/app/scripts/ingestion/sites/nersc.crontab.example -# -# Update REPO_DIR and SIMBOARD_API_TOKEN before use. For better secret hygiene, -# source the token from a protected env file instead of storing it directly in -# your crontab. - -SHELL=/bin/bash -PATH=/usr/local/bin:/usr/bin:/bin -CRON_TZ=UTC - -REPO_DIR=/path/to/simboard -SIMBOARD_API_TOKEN=replace-me -SIMBOARD_API_BASE_URL=https://simboard-dev-api.e3sm.org -MACHINE_NAME=perlmutter -DRY_RUN=true -PERF_ARCHIVE_ROOT=/global/cfs/projectdirs/e3sm/performance_archive -OLD_PERF_ARCHIVE_ROOT=/global/cfs/projectdirs/e3sm/OLD_PERF - -# Staging scan: run every hour. -0 * * * * cd ${REPO_DIR} && SCAN_MODE=staging ${REPO_DIR}/backend/app/scripts/ingestion/sites/nersc.sh >> ${REPO_DIR}/backend/app/scripts/ingestion/sites/nersc.sh.staging.log 2>&1 - -# Archive scan: run daily at 03:15 UTC. Add ARCHIVE_YEAR_START / ARCHIVE_YEAR_END -# here only when you want a scoped archive backfill. Values may use YYYY or YYYY-MM. -15 3 * * * cd ${REPO_DIR} && SCAN_MODE=archive ARCHIVE_YEAR_START=2025-01 ARCHIVE_YEAR_END=2025-03 ${REPO_DIR}/backend/app/scripts/ingestion/sites/nersc.sh >> ${REPO_DIR}/backend/app/scripts/ingestion/sites/nersc.sh.log 2>&1 - -# Diagnostics provenance scan: start dry-run, inspect logs, then set DRY_RUN=false. -20 * * * * cd ${REPO_DIR} && ${REPO_DIR}/backend/app/scripts/ingestion/sites/nersc-diagnostics-scanner.sh >> ${REPO_DIR}/backend/app/scripts/ingestion/sites/nersc-diagnostics-scanner.log 2>&1 - -# Chrysalis diagnostics provenance scan; use its local checkout path for REPO_DIR. -25 * * * * cd ${REPO_DIR} && MACHINE_NAME=chrysalis ${REPO_DIR}/backend/app/scripts/ingestion/sites/lcrc-diagnostics-scanner.sh >> ${REPO_DIR}/backend/app/scripts/ingestion/sites/lcrc-diagnostics-scanner.log 2>&1 diff --git a/backend/app/scripts/ingestion/sites/nersc.sh b/backend/app/scripts/ingestion/sites/nersc.sh deleted file mode 100755 index 9816bb77..00000000 --- a/backend/app/scripts/ingestion/sites/nersc.sh +++ /dev/null @@ -1,30 +0,0 @@ -#!/usr/bin/env bash - -set -euo pipefail - -SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" -BACKEND_DIR="$(cd -- "${SCRIPT_DIR}/../../../../" && pwd)" -PYTHON_BIN="${PYTHON_BIN:-${BACKEND_DIR}/.venv/bin/python}" - -if [[ ! -x "${PYTHON_BIN}" ]]; then - echo "Expected Python interpreter at ${PYTHON_BIN}" >&2 - echo "Run 'make install' from the repository root to create it." >&2 - exit 1 -fi - -: "${SIMBOARD_API_TOKEN:?SIMBOARD_API_TOKEN must be set before running this script.}" - -export SIMBOARD_API_BASE_URL="${SIMBOARD_API_BASE_URL:-https://simboard-dev-api.e3sm.org}" -export MACHINE_NAME="${MACHINE_NAME:-perlmutter}" -export SCAN_MODE="${SCAN_MODE:-archive}" -export DRY_RUN="${DRY_RUN:-true}" -export PERF_ARCHIVE_ROOT="${PERF_ARCHIVE_ROOT:-/global/cfs/projectdirs/e3sm/performance_archive}" -export OLD_PERF_ARCHIVE_ROOT="${OLD_PERF_ARCHIVE_ROOT:-/global/cfs/projectdirs/e3sm/OLD_PERF}" - -if [[ "${SCAN_MODE}" != "staging" && "${SCAN_MODE}" != "archive" ]]; then - echo "SCAN_MODE must be either 'staging' or 'archive'." >&2 - exit 1 -fi - -cd "${BACKEND_DIR}" -exec "${PYTHON_BIN}" -m app.scripts.ingestion.nersc_archive_ingestor "$@" diff --git a/backend/app/scripts/ingestion/sites/site_ingestion_launcher.sh b/backend/app/scripts/ingestion/sites/site_ingestion_launcher.sh new file mode 100755 index 00000000..be9fb341 --- /dev/null +++ b/backend/app/scripts/ingestion/sites/site_ingestion_launcher.sh @@ -0,0 +1,130 @@ +#!/usr/bin/env bash +set -euo pipefail + +if (( $# != 2 )) || [[ ! $1 =~ ^[a-z0-9_-]+$ ]] || [[ $2 != "archive" && $2 != "staging" ]]; then + echo "Usage: $0 " >&2 + exit 1 +fi + +site=$1 +scan_mode=$2 + +: "${SIMBOARD_ROOT:?SIMBOARD_ROOT must be defined in crontab or prior}" + +script_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" +site_config="${SIMBOARD_SITE_CONFIG:-${script_dir}/${site}.config}" + +if [[ ! -r "${site_config}" ]]; then + echo "Site configuration not readable: ${site_config}" >&2 + exit 1 +fi + +# These can be standardized, or overridden by specifiying in the site_config file. +export SIMBOARD_WORKDIR="${SIMBOARD_ROOT}/operations" +export SIMBOARD_MODULES="${SIMBOARD_ROOT}/repository/simboard/backend" + +# Site config provides paths, runner module, and optional authentication helpers. +# NOTE: config may override SIMBOARD_WORKDIR and SIMBOARD_MODULES, but need not. +source "${site_config}" + +echo "DEBUG: SIMBOARD_WORKDIR = ${SIMBOARD_WORKDIR}" >> TMPLOG 2>&1 +echo "DEBUG: SIMBOARD_MODULES = ${SIMBOARD_MODULES}" >> TMPLOG 2>&1 + +echo "SIMBOARD_ENV_FILE = $SIMBOARD_ENV_FILE" >> TMPLOG 2>&1 +echo "MACHINE_NAME = $MACHINE_NAME" >> TMPLOG 2>&1 + +: "${SIMBOARD_INGESTOR_MODULE:?SIMBOARD_INGESTOR_MODULE must be set by the site configuration}" + +export SCAN_MODE="${scan_mode}" + +# Site config supplies archive lower bound; callers may override it. +if [[ $scan_mode == "archive" ]]; then + export ARCHIVE_YEAR_START="${ARCHIVE_YEAR_START:-${SIMBOARD_DEFAULT_ARCHIVE_YEAR_START:?SIMBOARD_DEFAULT_ARCHIVE_YEAR_START must be set by the site configuration}}" +fi + +echo "SIMBOARD_API_TOKEN_FILE=${SIMBOARD_API_TOKEN_FILE}" >> TMPLOG 2>&1 +echo "SIMBOARD_INGESTOR_MODULE=${SIMBOARD_INGESTOR_MODULE}" >> TMPLOG 2>&1 +echo "SIMBOARD_API_BASE_URL=${SIMBOARD_API_BASE_URL}" >> TMPLOG 2>&1 +echo "SIMBOARD_DEFAULT_ARCHIVE_YEAR_START=${SIMBOARD_DEFAULT_ARCHIVE_YEAR_START}" >> TMPLOG 2>&1 + +# Optional max cases per run, default is no limit. +export MAX_CASES_PER_RUN="${MAX_CASES_PER_RUN:-1}" + +# Dry runs default to read-only remote-state validation. Set +# DRY_RUN_USE_REMOTE_STATE=false for credential-free offline scanning. +dry_run_normalized="${DRY_RUN:-true}" +dry_run_normalized="${dry_run_normalized#"${dry_run_normalized%%[![:space:]]*}"}" +dry_run_normalized="${dry_run_normalized%"${dry_run_normalized##*[![:space:]]}"}" +remote_state_normalized="${DRY_RUN_USE_REMOTE_STATE:-true}" +remote_state_normalized="${remote_state_normalized#"${remote_state_normalized%%[![:space:]]*}"}" +remote_state_normalized="${remote_state_normalized%"${remote_state_normalized##*[![:space:]]}"}" + +load_api_configuration() { + : "${SIMBOARD_ENV_FILE:?SIMBOARD_ENV_FILE must be set when remote API access is enabled}" + : "${SIMBOARD_API_TOKEN_FILE:?SIMBOARD_API_TOKEN_FILE must be set when remote API access is enabled}" + source "${SIMBOARD_ENV_FILE}" + source "${SIMBOARD_API_TOKEN_FILE}" + : "${SIMBOARD_API_BASE_URL:?SIMBOARD_API_BASE_URL must be set when remote API access is enabled}" + : "${SIMBOARD_API_TOKEN:?SIMBOARD_API_TOKEN failed to be set}" +} + +shopt -s nocasematch +case "${dry_run_normalized}" in + 0|false|no|off) + load_api_configuration + ;; + *) + case "${remote_state_normalized}" in + 0|false|no|off) ;; + *) load_api_configuration ;; + esac + ;; +esac +shopt -u nocasematch + +export PYTHON_BIN="${PYTHON_BIN:-${SIMBOARD_MODULES}/.venv/bin/python}" + +if [[ ! -d "${SIMBOARD_MODULES}/.venv" || ! -x "${PYTHON_BIN}" ]]; then + echo "Expected Python interpreter at ${PYTHON_BIN}" >&2 + echo "Run 'make install' from the repository root to create it." >&2 + exit 1 +fi + +echo "DEBUG_pre-activate: SIMBOARD_INGESTOR_MODULE = ${SIMBOARD_INGESTOR_MODULE}" >> TMPLOG 2>&1 +echo "DEBUG_pre-activate: MAX_CASES_PER_RUN = ${MAX_CASES_PER_RUN}" >> TMPLOG 2>&1 +echo "DEBUG_pre-activate: PYTHON_BIN = ${PYTHON_BIN}" >> TMPLOG 2>&1 +echo "DEBUG_pre-activate: SIMBOARD_ENV_FILE = ${SIMBOARD_ENV_FILE}" >> TMPLOG 2>&1 +echo "DEBUG_pre-activate: SIMBOARD_API_TOKEN_FILE = ${SIMBOARD_API_TOKEN_FILE}" >> TMPLOG 2>&1 +echo "DEBUG_pre-activate: SIMBOARD_API_BASE_URL = ${SIMBOARD_API_BASE_URL}" >> TMPLOG 2>&1 +printf 'DEBUG: ARCHIVE_YEAR_START=%q\n' "${ARCHIVE_YEAR_START-}" >> TMPLOG 2>&1 + +ts=`date -u +%Y%m%d_%H%M%S` +LOG_FILE="$SIMBOARD_WORKDIR/SBCS-$ts.log" +echo "DEBUG: LOG_FILE = ${LOG_FILE}" >> TMPLOG 2>&1 + +echo "DEBUG_post-activate: SIMBOARD_INGESTOR_MODULE = ${SIMBOARD_INGESTOR_MODULE}" >> ${LOG_FILE} 2>&1 +echo "DEBUG_post-activate: MAX_CASES_PER_RUN = ${MAX_CASES_PER_RUN}" >> ${LOG_FILE} 2>&1 +echo "DEBUG_post-activate: PYTHON_BIN = ${PYTHON_BIN}" >> ${LOG_FILE} 2>&1 +echo "DEBUG_post-activate: SIMBOARD_ENV_FILE = ${SIMBOARD_ENV_FILE}" >> ${LOG_FILE} 2>&1 +echo "DEBUG_post-activate: SIMBOARD_API_TOKEN_FILE = ${SIMBOARD_API_TOKEN_FILE}" >> ${LOG_FILE} 2>&1 +echo "DEBUG_post-activate: SIMBOARD_API_BASE_URL = ${SIMBOARD_API_BASE_URL}" >> ${LOG_FILE} 2>&1 +printf 'DEBUG: ARCHIVE_YEAR_START=%q\n' "${ARCHIVE_YEAR_START-}" >> ${LOG_FILE} 2>&1 + + +LOCK_FILE="$SIMBOARD_WORKDIR/SBCS.lock" +exec 200>"$LOCK_FILE" +if ! flock -n 200; then + echo "[$(date -Is)] SKIP launch simboard collection, lock already held, pid $$" >> "$LOG_FILE" + exit 0 +fi + +cleanup() { + echo "[$(date -Is)] simboard collection launcher exiting, pid $$" >> "$LOG_FILE" +} +trap cleanup EXIT + +# Run the app +cd "${SIMBOARD_MODULES}" +curdir=`pwd` +echo "DEBUG_post-activate: curdir = ${curdir}" >> ${LOG_FILE} 2>&1 +exec "${PYTHON_BIN}" -m "${SIMBOARD_INGESTOR_MODULE}" >> "$LOG_FILE" 2>&1 diff --git a/backend/docs/154-ingestions/codex-implementation-plan.md b/backend/docs/154-ingestions/codex-implementation-plan.md new file mode 100644 index 00000000..59322338 --- /dev/null +++ b/backend/docs/154-ingestions/codex-implementation-plan.md @@ -0,0 +1,168 @@ +# Chrysalis-First SimBoard HPC Ingestion Plan + +## Executive Summary + +Extend SimBoard metadata ingestion beyond Perlmutter/NERSC with one reusable +HPC ingestion framework and thin per-site adapters. Do not create separate +one-off implementations per site, and do not copy the NERSC Spin deployment +pattern unchanged. + +Chrysalis is the priority implementation target. It has the clearest current +path because the PACE reference and the existing GitHub script both indicate a +Sandia Jenkins workflow and a known archive root. + +Hold off on Compy, Aurora, and Frontier implementation until accounts or +equivalent native-runner access exist. Without access, implementation cannot +validate archive paths, scheduler behavior, token storage, network egress, or +metadata layout. These sites remain documented as future candidates only. + +Primary source: `/Users/vo13/Downloads/EPG-PACE Collection and Upload Reference-280426-174740.pdf`. + +Supporting GitHub sources: + +- `https://github.com/E3SM-Project/E3SM_test_scripts/blob/master/jenkins/chrysalis_pace.sh` +- `https://github.com/E3SM-Project/E3SM_test_scripts/blob/master/jenkins/compy_pace.sh` +- `https://github.com/E3SM-Project/E3SM_test_scripts/blob/master/util/pace_archive.sh` + +## Current State By Site + +| Site | Current state from source materials | SimBoard plan | +| --- | --- | --- | +| Perlmutter | Already implemented through NERSC Spin CronJob against NERSC-mounted archive storage. | Keep existing deployment. Use as reference for scanner, state, retry, dry-run, and logging behavior. | +| Chrysalis | PACE PDF and `chrysalis_pace.sh` show Sandia Jenkins and `/lcrc/group/e3sm/PERF_Chrysalis/performance_archive`. | Priority site. Add thin SimBoard shell wrapper for Jenkins and run shared Python ingestor. | +| Compy | PACE PDF and `compy_pace.sh` show Sandia Jenkins and `/compyfs/performance_archive`. | Defer until Compy account or native Jenkins validation exists. | +| Aurora | PACE PDF shows ALCF GitLab scheduled daily job and `/lus/flare/projects/E3SM_Dec/performance_archive`. | Defer until ALCF account/native-runner access exists. Do not force Jenkins. | +| Frontier | PACE PDF shows `/lustre/orion/proj-shared/cli115` and local cron/unknown frequency. | Defer until OLCF account, owner, runner, and script details are confirmed. | + +## Common Architecture Recommendation + +Use this common flow for all supported sites: + +```text +site scheduler -> thin SimBoard .sh wrapper -> shared Python ingestor -> SimBoard API +``` + +Thin site wrappers should live in SimBoard because SimBoard owns the API +contract, runtime environment variables, and ingestion behavior. Wrappers should +only: + +- load site-specific modules or Python environment when needed +- set `MACHINE_NAME` +- set `PERF_ARCHIVE_ROOT` +- set `STATE_PATH` +- require `SIMBOARD_API_BASE_URL` +- require `SIMBOARD_API_TOKEN` +- call `python -m app.scripts.ingestion.hpc_archive_ingestor` + +The shared Python ingestor should own: + +- archive scanning +- metadata validation +- idempotent state tracking +- dry-run behavior +- retry/backoff +- structured logs +- SimBoard API submission + +## Shared Components Vs Site-Specific Components + +Standardize these parts: + +- scan and parseable execution discovery +- metadata validation using existing SimBoard parser behavior +- state-file deduplication +- dry-run and capped-ingest controls +- retry/backoff and deterministic non-zero failure exits +- service-account token authentication +- structured startup, scan, candidate, success, failure, and summary logs + +Keep these parts site-specific: + +- scheduler: Jenkins, GitLab, cron, or site-native runner +- module/Python setup +- archive root and state path +- secret storage and token rotation workflow +- network/proxy/egress setup +- local filesystem permissions + +## Execution Model By Site + +| Site | Execution model | +| --- | --- | +| Perlmutter | Existing NERSC Spin CronJob. | +| Chrysalis | Sandia Jenkins wrapper. | +| Compy | Sandia Jenkins later, after account/native-runner validation. | +| Aurora | ALCF GitLab later, after ALCF access validation. | +| Frontier | Local cron or OLCF-native runner later, after owner/runtime confirmation. | + +## Reuse And Generalization Guidance + +Generalize the current NERSC ingestor by reusing its durable behavior: + +- archive scan +- execution-dir validation +- idempotent state +- dry-run mode +- retry/backoff +- structured logging + +Do not generalize by hardcoding NERSC assumptions: + +- no Perlmutter default in site wrappers +- no assumption that SimBoard can mount every remote DOE filesystem +- no assumption that every site uses NERSC Spin CronJob +- no single shared token across sites + +For Chrysalis, start with path-based ingestion only if the runtime can present a +path readable by the SimBoard ingestion API. If the SimBoard backend cannot read +the site filesystem, switch that site to upload-mode ingestion after access and +sample data validation. + +## Recommended Rollout Order + +1. Confirm with @rljacob that Chrysalis is the highest-value first site. +2. Preserve current Perlmutter/NERSC behavior. +3. Add generic `hpc_archive_ingestor` entrypoint that delegates to existing + scanner/state/retry/logging logic. +4. Add Chrysalis Jenkins wrapper. +5. Validate Chrysalis with dry-run and capped ingest. +6. Move Chrysalis to scheduled Jenkins only after state, counts, failure status, + token storage, and logs are verified. +7. Re-rank Compy, Aurora, and Frontier after accounts or native-runner access + are available. + +## Risks, Unknowns, And Assumptions + +Risks and unknowns: + +- Compy, Aurora, and Frontier cannot be safely implemented without access. +- Remote DOE filesystems may not be readable from the SimBoard backend. +- Non-NERSC sites may require upload-mode ingestion rather than path-mode + ingestion. +- Site egress to SimBoard may require proxy or firewall changes. +- Token storage and rotation are site-specific operational concerns. +- Existing PACE cleanup removes files larger than 50 MB; confirm this does not + remove metadata required by SimBoard. + +Assumptions: + +- Anvil is out of scope for this plan. +- Existing SimBoard `/ingestions/from-path` and `/ingestions/from-upload` APIs + are sufficient for initial rollout planning. +- One service-account token should be provisioned per site. +- Existing PACE scripts remain responsible for PACE collection/upload. SimBoard + wrappers only bridge archived metadata into SimBoard. + +## Concrete Next Steps + +1. Ask @rljacob to confirm Chrysalis as first priority. +2. Confirm Chrysalis Jenkins owner, token storage mechanism, and service account + rotation path. +3. Confirm Chrysalis archive root: + `/lcrc/group/e3sm/PERF_Chrysalis/performance_archive`. +4. Get one recent Chrysalis archived case path. +5. Run Chrysalis dry-run with `DRY_RUN=true`. +6. Run capped ingest with `MAX_CASES_PER_RUN`. +7. Verify SimBoard created/duplicate/error counts. +8. Verify state file prevents repeat ingestion. +9. Verify Jenkins marks ingestion failures as failed jobs. diff --git a/backend/docs/154-ingestions/ingestion-sites-onboarding.md b/backend/docs/154-ingestions/ingestion-sites-onboarding.md new file mode 100644 index 00000000..d32d28f4 --- /dev/null +++ b/backend/docs/154-ingestions/ingestion-sites-onboarding.md @@ -0,0 +1,127 @@ +# Site Ingestion Onboarding + +## Purpose + +This work extends SimBoard metadata ingestion beyond Perlmutter/NERSC. The first target is Chrysalis because it already has a Jenkins-based PACE workflow and can use the same ingestion model with only site-specific environment defaults. + +Tracking issue: https://github.com/E3SM-Project/simboard/issues/154 + +Takeover PR: https://github.com/E3SM-Project/simboard/pull/169 + +Current branch: + +```bash +feature/154-ingestion-sites +``` + +## Current State + +The repository uses a config-driven host-side launcher for site collection: + +- `backend/app/scripts/ingestion/sites/site_ingestion_launcher.sh` loads a named site config and selects its ingestion runner. +- `backend/app/scripts/ingestion/sites/chrysalis.config` and `nersc.config` provide site-specific paths, machine names, runner modules, and protected credential-file locations. +- `backend/app/scripts/ingestion/hpc_upload_archive_ingestor.py` is the scheduler-agnostic upload runner for remote HPC sites; `nersc_archive_ingestor.py` handles NERSC path ingestion. +- `backend/app/scripts/README.md` documents launcher use and site-config responsibilities. +- `backend/tests/features/ingestion/test_site_collection_launcher.py` covers config-driven offline launcher behavior. + +PR 169 is an open draft for this Chrysalis work. Take it over from there rather than starting a new branch. The PR currently notes that local validation was blocked because PostgreSQL was unavailable at `127.0.0.1`, so backend tests still need to be rerun in a working local or CI environment. + +The key design rule is that shell wrappers should stay thin. Put reusable ingestion behavior in Python, not in site-specific shell scripts. + +## How Ingestion Works + +The existing ingestor scans a performance archive directory, finds parseable execution directories, tracks state, and calls the SimBoard path-ingestion API for changed cases. + +The shared entrypoint is intended to be stable across schedulers: + +```bash +app/scripts/ingestion/sites/site_ingestion_launcher.sh chrysalis staging +``` + +Site configs should set only local defaults such as: + +- `MACHINE_NAME` +- `PERF_ARCHIVE_ROOT` +- `DRY_RUN` +- `DRY_RUN_USE_REMOTE_STATE` +- `SIMBOARD_INGESTOR_MODULE` +- protected environment and token file locations + +For normal execution and default dry runs, the launcher loads the API environment +and token from the protected files referenced by the site config: + +- `SIMBOARD_API_BASE_URL` +- `SIMBOARD_API_TOKEN` + +See `docs/hpc_api_token_authentication.md` for service account and API token setup. + +## Chrysalis Handoff + +Start with Chrysalis. + +Current launcher invocation: + +```bash +backend/app/scripts/ingestion/sites/site_ingestion_launcher.sh chrysalis staging +``` + +The Chrysalis config defaults to: + +- `MACHINE_NAME=chrysalis` +- `PERF_ARCHIVE_ROOT=/lcrc/group/e3sm/PERF_Chrysalis/performance_archive` +- `DRY_RUN=true` + +Set `DRY_RUN_USE_REMOTE_STATE=false` only when a credential-free offline scan is +needed; default dry runs read remote state and checkpoints without writing data. + +Before enabling real ingestion, validate: + +- The archive path exists and is readable from the Jenkins runtime. +- Jenkins can run the backend Python environment. +- Jenkins can inject `SIMBOARD_API_BASE_URL` and `SIMBOARD_API_TOKEN` without logging the token. +- The Jenkins host has network egress to the SimBoard API. +- Dry-run output shows expected candidate counts. +- The state file location is writable and persists across runs. + +Do not set `DRY_RUN=false` until the dry-run behavior has been reviewed. + +## Recommended Task Order + +1. Review the current branch implementation and confirm it matches the thin-wrapper design. +2. Rerun backend tests for PR 169 in an environment with PostgreSQL available. +3. Validate the Chrysalis archive path and Jenkins environment. +4. Create or identify the SimBoard service account and API token for HPC ingestion. +5. Configure Jenkins to provide `SIMBOARD_API_BASE_URL` and `SIMBOARD_API_TOKEN` securely. +6. Run the Chrysalis launcher with the default dry-run mode. +7. Review candidate counts, skipped cases, errors, and state-file behavior. +8. Enable non-dry-run ingestion only after validation. +9. Apply the same wrapper pattern to additional sites once access is available. + +## Remaining Sites + +Priority and status from issue discussion: + +- Chrysalis: first target; Jenkins workflow. +- Frontier: request or confirm account access. +- Aurora: request or confirm account access. +- Compy: request or confirm account access. +- Anvil: removed from scope. + +Expected runners from the PACE references: + +- Chrysalis and Compy use Jenkins. +- Frontier uses cron. +- Aurora uses ALCF GitLab. + +Confirm these runner assumptions before implementing wrappers for non-Chrysalis sites. + +## References + +- Issue 154: https://github.com/E3SM-Project/simboard/issues/154 +- PR 169: https://github.com/E3SM-Project/simboard/pull/169 +- PACE overview: https://e3sm.atlassian.net/wiki/spaces/EPG/pages/776437853/Performance+Analytics+for+Computational+Experiments+PACE +- PACE collection/upload reference: https://e3sm.atlassian.net/wiki/spaces/EPG/pages/5477335106/PACE+Collection+and+Upload+Reference +- Existing site script wrappers: https://github.com/E3SM-Project/E3SM_test_scripts/tree/master/jenkins +- Existing PACE archive script: https://github.com/E3SM-Project/E3SM_test_scripts/blob/master/util/pace_archive.sh +- SimBoard script docs: `backend/app/scripts/README.md` +- API token docs: `docs/hpc_api_token_authentication.md` diff --git a/backend/docs/154-ingestions/simboard-hpc-ingestion-architecture.png b/backend/docs/154-ingestions/simboard-hpc-ingestion-architecture.png new file mode 100644 index 00000000..4e364fb0 Binary files /dev/null and b/backend/docs/154-ingestions/simboard-hpc-ingestion-architecture.png differ diff --git a/backend/tests/features/ingestion/test_archive_workflow.py b/backend/tests/features/ingestion/test_archive_workflow.py index a0ad72d2..1b6a62a6 100644 --- a/backend/tests/features/ingestion/test_archive_workflow.py +++ b/backend/tests/features/ingestion/test_archive_workflow.py @@ -74,8 +74,16 @@ def log_event(event: str, fields: dict[str, Any] | None = None) -> None: archive_root = tmp_path / "archive" archive_root.mkdir() - config = replace(_config(archive_root), api_token="") - assert not _validate_run_preconditions(config, log_event_fn=log_event) + default_dry_run_config = replace(_config(archive_root, dry_run=True), api_token="") + assert not _validate_run_preconditions( + default_dry_run_config, log_event_fn=log_event + ) + + dry_run_config = replace(default_dry_run_config, dry_run_use_remote_state=False) + assert _validate_run_preconditions(dry_run_config, log_event_fn=log_event) + + ingest_config = replace(_config(archive_root), api_token="") + assert not _validate_run_preconditions(ingest_config, log_event_fn=log_event) assert logged_events[-1] == ( "configuration_error", {"error": "SIMBOARD_API_TOKEN is required"}, @@ -129,6 +137,7 @@ def fake_log_event(event: str, fields: dict[str, Any] | None = None) -> None: { "machine_name": "pm", "dry_run": True, + "dry_run_use_remote_state": True, "max_cases_per_run": 5, "max_attempts": 2, "request_timeout_seconds": 60, diff --git a/backend/tests/features/ingestion/test_hpc_upload_archive_ingestor.py b/backend/tests/features/ingestion/test_hpc_upload_archive_ingestor.py index f329b3cb..f95d1372 100644 --- a/backend/tests/features/ingestion/test_hpc_upload_archive_ingestor.py +++ b/backend/tests/features/ingestion/test_hpc_upload_archive_ingestor.py @@ -610,18 +610,32 @@ def fake_discovery_post(*args, **kwargs): monkeypatch.setattr( upload_ingestor_module, "_fetch_ingestion_state", - lambda *args, **kwargs: _fresh_state(), + lambda *_args, **_kwargs: pytest.fail("dry run must not fetch API state"), + ) + monkeypatch.setattr( + upload_ingestor_module, + "_build_endpoint_url", + lambda *_: pytest.fail("dry run must not build upload endpoint"), + ) + monkeypatch.setattr( + upload_ingestor_module, + "_fetch_archive_checkpoints", + lambda *_args, **_kwargs: pytest.fail( + "dry run must not fetch archive checkpoints" + ), ) config = IngestorConfig( - api_base_url="http://backend:8000", - api_token="token", + api_base_url="", + api_token="", archive_root=archive_root, machine_name="perlmutter", dry_run=True, + dry_run_use_remote_state=False, max_cases_per_run=None, max_attempts=1, request_timeout_seconds=30, + scan_mode="archive", ) exit_code = _run_ingestor( @@ -785,7 +799,7 @@ def fake_log_event(event: str, fields: dict[str, object] | None = None) -> None: assert any(event == "state_fetch_failed" for event, _ in logged_events) -def test_run_ingestor_fetches_archive_checkpoints_before_state( +def test_run_ingestor_default_dry_run_fetches_archive_checkpoints_before_state( tmp_path: Path, monkeypatch, ) -> None: @@ -863,7 +877,7 @@ def fetch_state(*args: Any, **kwargs: Any) -> dict[str, Any]: api_token="token", archive_root=archive_root, machine_name="perlmutter", - dry_run=True, + dry_run=False, max_cases_per_run=None, max_attempts=1, request_timeout_seconds=30, diff --git a/backend/tests/features/ingestion/test_nersc_archive_ingestor.py b/backend/tests/features/ingestion/test_nersc_archive_ingestor.py index b9738bc4..cc1afaf7 100644 --- a/backend/tests/features/ingestion/test_nersc_archive_ingestor.py +++ b/backend/tests/features/ingestion/test_nersc_archive_ingestor.py @@ -456,7 +456,7 @@ def fake_post_request( assert captured_processed_execution_ids == [["101.1-1"]] -def test_run_ingestor_dry_run_without_token_returns_config_error( +def test_run_ingestor_dry_run_without_api_configuration_scans_offline( tmp_path: Path, monkeypatch, ) -> None: @@ -469,16 +469,35 @@ def fake_log_event(event: str, fields: dict[str, Any] | None = None) -> None: monkeypatch.setattr(ingestor_module, "_log_event", fake_log_event) monkeypatch.setattr(discovery_module, "_log_event", fake_log_event) + monkeypatch.setattr( + ingestor_module, + "_build_endpoint_url", + lambda *_: pytest.fail("dry run must not build ingestion endpoint"), + ) + monkeypatch.setattr( + ingestor_module, + "_fetch_ingestion_state", + lambda *_args, **_kwargs: pytest.fail("dry run must not fetch API state"), + ) + monkeypatch.setattr( + ingestor_module, + "_fetch_archive_checkpoints", + lambda *_args, **_kwargs: pytest.fail( + "dry run must not fetch archive checkpoints" + ), + ) config = IngestorConfig( - api_base_url="http://backend:8000", + api_base_url="", api_token="", archive_root=archive_root, machine_name="perlmutter", dry_run=True, + dry_run_use_remote_state=False, max_cases_per_run=None, max_attempts=1, request_timeout_seconds=30, + scan_mode="archive", ) exit_code = _run_ingestor( @@ -487,8 +506,8 @@ def fake_log_event(event: str, fields: dict[str, Any] | None = None) -> None: sleep_fn=lambda *_: None, ) - assert exit_code == 1 - assert any(event == "configuration_error" for event, _ in logged_events) + assert exit_code == 0 + assert any(event == "dry_run_completed" for event, _ in logged_events) def test_run_ingestor_without_token_returns_config_error( @@ -562,7 +581,7 @@ def fake_log_event(event: str, fields: dict[str, Any] | None = None) -> None: assert not any(event == "scan_completed" for event, _ in logged_events) -def test_run_ingestor_fetches_state_before_archive_checkpoints( +def test_run_ingestor_default_dry_run_fetches_state_before_archive_checkpoints( tmp_path: Path, monkeypatch, ) -> None: @@ -624,7 +643,7 @@ def test_run_ingestor_returns_failure_when_checkpoint_fetch_fails( api_token="token", archive_root=archive_root, machine_name="perlmutter", - dry_run=True, + dry_run=False, max_cases_per_run=None, max_attempts=1, request_timeout_seconds=30, @@ -821,6 +840,7 @@ def test_build_config_from_env_parses_valid_values(monkeypatch, tmp_path: Path) assert config.archive_root == (tmp_path / "archive").resolve() assert config.machine_name == "pm" assert config.dry_run is True + assert config.dry_run_use_remote_state is True assert config.max_cases_per_run == 5 assert config.max_attempts == 4 assert config.request_timeout_seconds == 90 @@ -846,6 +866,18 @@ def test_build_config_from_env_parses_archive_mode_and_year_range( assert config.archive_year_end == "2025-12" +def test_build_config_from_env_allows_offline_dry_runs( + monkeypatch, tmp_path: Path +) -> None: + monkeypatch.setenv("PERF_ARCHIVE_ROOT", str(tmp_path / "archive")) + monkeypatch.setenv("DRY_RUN_USE_REMOTE_STATE", "false") + + config = _build_config_from_env() + + assert config.dry_run is True + assert config.dry_run_use_remote_state is False + + def test_build_config_from_env_parses_archive_mode_and_month_range( monkeypatch, tmp_path: Path ) -> None: @@ -1005,6 +1037,20 @@ def test_module_main_guard_exits_via_system_exit_on_configuration_error( assert exc_info.value.code == 1 +def test_generic_hpc_module_main_guard_delegates_to_ingestor(monkeypatch) -> None: + script_path = ( + Path(__file__).resolve().parents[3] + / "app/scripts/ingestion/hpc_upload_archive_ingestor.py" + ) + monkeypatch.setenv("MAX_ATTEMPTS", "0") + monkeypatch.setattr(logging.Logger, "info", lambda *args, **kwargs: None) + + with pytest.raises(SystemExit) as exc_info: + runpy.run_path(str(script_path), run_name="__main__") + + assert exc_info.value.code == 1 + + @pytest.mark.parametrize( ("value", "default", "expected"), [ diff --git a/backend/tests/features/ingestion/test_site_collection_launcher.py b/backend/tests/features/ingestion/test_site_collection_launcher.py new file mode 100644 index 00000000..75dd5a5e --- /dev/null +++ b/backend/tests/features/ingestion/test_site_collection_launcher.py @@ -0,0 +1,156 @@ +"""Tests for the config-driven host-side site collection launcher.""" + +import os +import shlex +import subprocess +from pathlib import Path + + +def _launcher_path() -> Path: + return ( + Path(__file__).resolve().parents[3] + / "app/scripts/ingestion/sites/site_ingestion_launcher.sh" + ) + + +def _write_executable(path: Path, contents: str) -> Path: + path.write_text(contents, encoding="utf-8") + path.chmod(0o755) + return path + + +def test_launcher_runs_configured_ingestor_offline(tmp_path: Path) -> None: + work_dir = tmp_path / "work" + work_dir.mkdir() + capture_path = tmp_path / "environment.txt" + bin_dir = tmp_path / "bin" + bin_dir.mkdir() + fake_python = _write_executable( + bin_dir / "python", + "#!/usr/bin/env bash\n" + 'printf "%s\\n" "${SCAN_MODE}" "${ARCHIVE_YEAR_START-unset}" ' + '"${MACHINE_NAME}" "$*" > "${CAPTURE_PATH}"\n', + ) + _write_executable(bin_dir / "flock", "#!/usr/bin/env bash\nexit 0\n") + backend_dir = Path(__file__).resolve().parents[3] + site_config = tmp_path / "test.config" + site_config.write_text( + "\n".join( + [ + f"export SIMBOARD_REPODIR={shlex.quote(str(backend_dir))}", + f"export SIMBOARD_WORKDIR={shlex.quote(str(work_dir))}", + "export SIMBOARD_INGESTOR_MODULE=app.scripts.ingestion.nersc_archive_ingestor", + "export SIMBOARD_DEFAULT_ARCHIVE_YEAR_START=2024-01", + "export MACHINE_NAME=test-machine", + "export DRY_RUN=true", + "export DRY_RUN_USE_REMOTE_STATE=false", + f"export PYTHON_BIN={shlex.quote(str(fake_python))}", + ] + ) + + "\n", + encoding="utf-8", + ) + env = os.environ.copy() + env.pop("SIMBOARD_API_BASE_URL", None) + env.pop("SIMBOARD_API_TOKEN", None) + env["CAPTURE_PATH"] = str(capture_path) + env["SIMBOARD_SITE_CONFIG"] = str(site_config) + env["PATH"] = f"{bin_dir}:{env['PATH']}" + + result = subprocess.run( + [_launcher_path(), "test", "archive"], + capture_output=True, + check=False, + env=env, + text=True, + ) + + assert result.returncode == 0, result.stderr + assert capture_path.read_text(encoding="utf-8").splitlines() == [ + "archive", + "2024-01", + "test-machine", + "-m app.scripts.ingestion.nersc_archive_ingestor", + ] + + +def test_launcher_loads_credentials_for_default_remote_state_dry_run( + tmp_path: Path, +) -> None: + work_dir = tmp_path / "work" + work_dir.mkdir() + bin_dir = tmp_path / "bin" + bin_dir.mkdir() + fake_python = _write_executable(bin_dir / "python", "#!/usr/bin/env bash\nexit 0\n") + _write_executable(bin_dir / "flock", "#!/usr/bin/env bash\nexit 0\n") + environment_file = tmp_path / "environment.sh" + environment_file.write_text( + "export SIMBOARD_API_BASE_URL=https://example.test\n", encoding="utf-8" + ) + token_file = tmp_path / "token.sh" + token_file.write_text("export SIMBOARD_API_TOKEN=test-token\n", encoding="utf-8") + backend_dir = Path(__file__).resolve().parents[3] + site_config = tmp_path / "test.config" + site_config.write_text( + "\n".join( + [ + f"export SIMBOARD_REPODIR={shlex.quote(str(backend_dir))}", + f"export SIMBOARD_WORKDIR={shlex.quote(str(work_dir))}", + "export SIMBOARD_INGESTOR_MODULE=app.scripts.ingestion.nersc_archive_ingestor", + "export SIMBOARD_DEFAULT_ARCHIVE_YEAR_START=2024-01", + f"export SIMBOARD_ENV_FILE={shlex.quote(str(environment_file))}", + f"export SIMBOARD_API_TOKEN_FILE={shlex.quote(str(token_file))}", + "export DRY_RUN=true", + f"export PYTHON_BIN={shlex.quote(str(fake_python))}", + ] + ) + + "\n", + encoding="utf-8", + ) + env = os.environ.copy() + env.pop("SIMBOARD_API_BASE_URL", None) + env.pop("SIMBOARD_API_TOKEN", None) + env["SIMBOARD_SITE_CONFIG"] = str(site_config) + env["PATH"] = f"{bin_dir}:{env['PATH']}" + + result = subprocess.run( + [_launcher_path(), "test", "archive"], + capture_output=True, + check=False, + env=env, + text=True, + ) + + assert result.returncode == 0, result.stderr + + +def test_site_configs_define_their_ingestors() -> None: + sites_dir = _launcher_path().parent + + for config_name, expected_module, expected_machine in ( + ( + "nersc.config", + "app.scripts.ingestion.nersc_archive_ingestor", + "perlmutter", + ), + ( + "chrysalis.config", + "app.scripts.ingestion.hpc_upload_archive_ingestor", + "chrysalis", + ), + ): + result = subprocess.run( + [ + "bash", + "-c", + 'source "$1"; printf "%s\\n%s\\n" "$SIMBOARD_INGESTOR_MODULE" "$MACHINE_NAME"', + "bash", + str(sites_dir / config_name), + ], + capture_output=True, + check=False, + text=True, + ) + + assert result.returncode == 0, result.stderr + assert result.stdout.splitlines() == [expected_module, expected_machine] diff --git a/docs/README.md b/docs/README.md index a9abca83..7e1b416e 100644 --- a/docs/README.md +++ b/docs/README.md @@ -27,3 +27,4 @@ Use this directory as the documentation router by audience. - Deployment reference: [deploy/deployment-and-release.md](deploy/deployment-and-release.md) - NERSC Spin runbook: [deploy/nersc-spin-runbook.md](deploy/nersc-spin-runbook.md) - HPC token and service-account auth: [deploy/hpc-api-token-authentication.md](deploy/hpc-api-token-authentication.md) +- Read-only database access: [deploy/read-only-database-access.md](deploy/read-only-database-access.md) diff --git a/docs/architecture/metadata-ingestion.md b/docs/architecture/metadata-ingestion.md index 19947953..a01e2b8f 100644 --- a/docs/architecture/metadata-ingestion.md +++ b/docs/architecture/metadata-ingestion.md @@ -301,7 +301,8 @@ flowchart TD ### Runner Configuration -All automated ingestion requests require a bearer API token. Both site-side runners use: +Automated ingestion and remote-state dry runs require a bearer API token. Both +site-side runners use: - `SIMBOARD_API_BASE_URL` - `SIMBOARD_API_TOKEN` @@ -310,6 +311,14 @@ All automated ingestion requests require a bearer API token. Both site-side runn - `OLD_PERF_ARCHIVE_ROOT` - `MACHINE_NAME` - `DRY_RUN` +- `DRY_RUN_USE_REMOTE_STATE` + +With `DRY_RUN=true`, runners read remote ingestion state and archive checkpoints by +default, then scan and report candidates without writes. This requires +`SIMBOARD_API_BASE_URL` and `SIMBOARD_API_TOKEN`; the report excludes executions +already ingested remotely and snapshots already checkpointed. Set +`DRY_RUN_USE_REMOTE_STATE=false` for an offline scan using empty local state and no +API requests. They also support these tuning options: @@ -321,8 +330,10 @@ They also support these tuning options: `SCAN_MODE` selects whether a runner scans staging or archive roots. In archive mode, runners traverse only top-level `YYYY-MM` buckets under the configured -archive root. Year-range filters apply only to archive mode and are intended -for targeted backfills, not for normal staging collection. +archive root. Year-range filters apply only to archive mode. Direct Python +entrypoints leave both bounds unset; the NERSC and Chrysalis site wrappers +default `ARCHIVE_YEAR_START=2025-01` and leave `ARCHIVE_YEAR_END` unset. Callers +may override either bound for a differently scoped archive scan. `MAX_CASES_PER_RUN` is an optional per-run throttle. Leave it unset for normal operation when runners should submit every submission-qualified case they find. @@ -339,6 +350,21 @@ production-like archive at the target site. Record the observed archive creation upload, and retry timings from the events above before rollout; this document does not supply synthetic benchmark measurements. +### Operational Validation + +The runners use composable configuration rather than numbered dry-run modes: + +- Set `DRY_RUN=true` to scan and report candidates without persisted state changes; + it reads remote state by default. Set `DRY_RUN_USE_REMOTE_STATE=false` to avoid + API requests for an offline scan. +- Set `DRY_RUN=false` and a small `MAX_CASES_PER_RUN` value to validate live + ingestion on a bounded batch. This is real ingestion and persists results. +- Set `DRY_RUN=false` with no per-run cap for normal scheduled ingestion. + +Use one-off diagnostic scripts only for exceptional, data-specific +investigations. Do not add a permanent runner mode unless that workflow becomes +a recurring operational need. + ### Stored Results After ingestion, SimBoard stores normalized cases, executions, machines, artifacts, links, and audit records in PostgreSQL. Execution records preserve parsed `CASE_HASH` values so the frontend can group related executions inside a case without assigning a persistent reference execution. The frontend reads the resulting catalog data through `/api/v1` endpoints. diff --git a/docs/deploy/README.md b/docs/deploy/README.md index 049a953e..7b706154 100644 --- a/docs/deploy/README.md +++ b/docs/deploy/README.md @@ -5,9 +5,11 @@ Recommended reading order: 1. [Deployment and Release Guide](deployment-and-release.md) 2. [NERSC Spin Runbook](nersc-spin-runbook.md) 3. [HPC API Token Authentication](hpc-api-token-authentication.md) +4. [Read-Only Database Access](read-only-database-access.md) | Document | Purpose | | --------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- | | [Deployment and Release Guide](deployment-and-release.md) | CI/CD, image tagging, release rollout, migrations, rollback, and deployment troubleshooting. | | [NERSC Spin Runbook](nersc-spin-runbook.md) | NERSC Spin and Rancher operational setup, workload configuration, secrets, ingress, and service management. | | [HPC API Token Authentication](hpc-api-token-authentication.md) | Token-based authentication for automated HPC ingestion jobs. | +| [Read-Only Database Access](read-only-database-access.md) | SSH-tunneled PostgreSQL access for the `simboard_readonly` account. | diff --git a/docs/deploy/nersc-spin-runbook.md b/docs/deploy/nersc-spin-runbook.md index 8bd16da3..197a88e6 100644 --- a/docs/deploy/nersc-spin-runbook.md +++ b/docs/deploy/nersc-spin-runbook.md @@ -351,12 +351,12 @@ snapshots in any eligible month are discovered automatically. 4. **Create/update CronJob `nersc-staging-ingestor`** - Use the **Staging CronJob** section below. - Configure secret-backed environment variables from `nersc-staging-ingestor-env`. - - Keep the CronJob command on `python -m app.scripts.ingestion.nersc_archive_ingestor`. Do not switch this workload to `app/scripts/ingestion/sites/nersc.sh`; that wrapper is for host-side NERSC cron usage and defaults to host filesystem paths and API values that do not match this Spin workload. + - Keep the CronJob command on `python -m app.scripts.ingestion.nersc_archive_ingestor`. Do not switch this workload to `app/scripts/ingestion/sites/site_ingestion_launcher.sh`; that launcher is for host-side NERSC cron usage and loads host filesystem paths and API configuration that do not match this Spin workload. 5. **Create/update CronJob `nersc-archive-ingestor`** - Use the **Archive CronJob** section below. - Configure secret-backed environment variables from `nersc-archive-ingestor-env`. - - Keep the CronJob command on `python -m app.scripts.ingestion.nersc_archive_ingestor`. Do not switch this workload to `app/scripts/ingestion/sites/nersc.sh`; that wrapper is for host-side NERSC cron usage and defaults to host filesystem paths and API values that do not match this Spin workload. + - Keep the CronJob command on `python -m app.scripts.ingestion.nersc_archive_ingestor`. Do not switch this workload to `app/scripts/ingestion/sites/site_ingestion_launcher.sh`; that launcher is for host-side NERSC cron usage and loads host filesystem paths and API configuration that do not match this Spin workload. 6. **Validate both jobs once with dry run** - Set `DRY_RUN=true` in both ingestion secrets. diff --git a/docs/deploy/read-only-database-access.md b/docs/deploy/read-only-database-access.md new file mode 100644 index 00000000..9d9bf9c2 --- /dev/null +++ b/docs/deploy/read-only-database-access.md @@ -0,0 +1,226 @@ +# Read-Only Database Access + +Use this guide to connect to the SimBoard development PostgreSQL database with +the `simboard_readonly` account. The account is intended for inspecting data; +it must not be used by the application, migrations, or ingestion jobs. + +## Prerequisites + +- Authorization to use the `simboard_readonly` account and its password. +- SSH access to Perlmutter with your own NERSC account. +- A PostgreSQL client. Any client that can connect to PostgreSQL over a local + TCP port will work. + +The database endpoint is internal to NERSC Spin. Reach it through Perlmutter; +do not attempt to expose the database service publicly. + +## Connection details + +| Setting | Value | +| --- | --- | +| Database | `simboard` | +| Database user | `simboard_readonly` | +| Internal database host | `db-loadbalancer.simboard.development.svc.spin.nersc.org` | +| Internal database port | `5432` | +| SSH host | `perlmutter-p1.nersc.gov` | +| SSH port | `22` | + +Use the password provided through an approved private channel. Do not store it +in this repository, a shared document, or a connection profile that is not +protected by your operating-system account. + +## Option 1: Create an SSH tunnel (works with any client) + +On your workstation, leave this command running while you use your database +client. Replace `YOUR_NERSC_USERNAME` with your NERSC login name. + +```bash +ssh -N -L 15432:db-loadbalancer.simboard.development.svc.spin.nersc.org:5432 \ + YOUR_NERSC_USERNAME@perlmutter-p1.nersc.gov +``` + +Then configure the database client to connect to: + +| Client setting | Value | +| --- | --- | +| Host | `127.0.0.1` | +| Port | `15432` | +| Database | `simboard` | +| Username | `simboard_readonly` | +| Password | Password supplied for this account | + +Keep the SSH session open for the entire database session. To stop access, +close the SSH session. + +For example, after opening the tunnel, `psql` can connect with: + +```bash +psql -h 127.0.0.1 -p 15432 -U simboard_readonly -d simboard +``` + +`psql` will prompt for the database password. + +## Option 2: Use your client’s SSH tunnel feature + +Many database clients can create the same tunnel themselves. Configure the +PostgreSQL connection using the **internal database host** and port from the +table above, then enable the client’s SSH tunnel/proxy option with: + +| SSH setting | Value | +| --- | --- | +| SSH host | `perlmutter-p1.nersc.gov` | +| SSH port | `22` | +| SSH username | Your NERSC username | +| Authentication | Your normal approved NERSC SSH method | + +Set the database username to `simboard_readonly` and enter the separately +provided database password. The screenshots shared with this guide show these +same two groups of settings in DBeaver, but the values apply to any client with +SSH tunneling support. + +## Verify the connection + +Run the following query after connecting: + +```sql +SELECT current_user, current_database(); +``` + +It should return `simboard_readonly` and `simboard`. Reads, such as `SELECT` +queries, should succeed for granted tables. Commands that change data or +schema, such as `INSERT`, `UPDATE`, `DELETE`, `CREATE`, or `ALTER`, should be +denied. + +## Getting started: ingestion and operations queries + +The `ingestions` table is the audit record for each upload or HPC-path +ingestion. Join it to `machines` to see where the data came from, and to +`executions` and `cases` to inspect the records it created. The queries below +are read-only and use date filters or `LIMIT` to keep exploratory queries +small. + +### Recent ingestion activity + +Use this first to see the most recent ingestion attempts and their outcomes. + +```sql +SELECT + i.created_at, + m.name AS machine, + i.source_type, + i.status, + i.created_count, + i.duplicate_count, + i.error_count, + i.source_reference +FROM ingestions AS i +JOIN machines AS m ON m.id = i.machine_id +ORDER BY i.created_at DESC +LIMIT 25; +``` + +`success`, `partial`, and `failed` are the ingestion status values. A +non-zero `error_count` or a `partial`/`failed` status is worth investigating. + +### Recent incomplete or failed ingestions + +This narrows the audit trail to outcomes that may need attention. + +```sql +SELECT + i.created_at, + m.name AS machine, + i.status, + i.error_count, + i.created_count, + i.duplicate_count, + i.source_type, + i.source_reference +FROM ingestions AS i +JOIN machines AS m ON m.id = i.machine_id +WHERE i.status IN ('partial', 'failed') + OR i.error_count > 0 +ORDER BY i.created_at DESC +LIMIT 50; +``` + +### Daily ingestion volume by machine + +Use this operational summary to look for gaps, spikes, or a rise in errors. + +```sql +SELECT + date_trunc('day', i.created_at) AS day, + m.name AS machine, + COUNT(*) AS ingestion_count, + SUM(i.created_count) AS executions_created, + SUM(i.duplicate_count) AS duplicates, + SUM(i.error_count) AS errors +FROM ingestions AS i +JOIN machines AS m ON m.id = i.machine_id +WHERE i.created_at >= CURRENT_TIMESTAMP - INTERVAL '14 days' +GROUP BY day, m.name +ORDER BY day DESC, m.name; +``` + +Change the interval to suit the investigation. For example, use `7 days` for +a shorter operational view. + +### Executions created by recent ingestions + +This joins ingestion audit records to their resulting execution and case +records. It is useful when an ingestion count looks unexpected. + +```sql +SELECT + i.created_at AS ingested_at, + m.name AS machine, + i.status AS ingestion_status, + c.name AS case_name, + e.execution_id, + e.status AS execution_status, + e.compset, + e.grid_name +FROM ingestions AS i +JOIN machines AS m ON m.id = i.machine_id +JOIN executions AS e ON e.ingestion_id = i.id +JOIN cases AS c ON c.id = e.case_id +ORDER BY i.created_at DESC, c.name, e.execution_id +LIMIT 100; +``` + +To investigate one ingestion, add `WHERE i.id = 'INGESTION_UUID'` before the +`ORDER BY` clause, replacing `INGESTION_UUID` with the identifier shown by +your client. + +### Current execution status overview + +This gives a compact view of the catalog's execution states by machine. + +```sql +SELECT + m.name AS machine, + e.status AS execution_status, + COUNT(*) AS execution_count +FROM executions AS e +JOIN cases AS c ON c.id = e.case_id +JOIN machines AS m ON m.id = c.machine_id +GROUP BY m.name, e.status +ORDER BY m.name, e.status; +``` + +Execution statuses include `created`, `queued`, `running`, `failed`, and +`completed`; `unknown` is used when no more specific state is available. + +## Troubleshooting + +- **SSH authentication fails:** confirm that you can SSH to Perlmutter with + the same NERSC account before configuring the database client. +- **Connection refused on `127.0.0.1:15432`:** confirm the SSH tunnel is still + running and that no other local program is using port `15432`. +- **Database authentication fails:** verify that the database username is + exactly `simboard_readonly` and obtain a fresh password from the account + administrator if necessary. +- **Database host cannot be resolved locally:** this is expected. The internal + hostname is resolved from Perlmutter through the SSH tunnel, not from your + workstation. diff --git a/mkdocs.yml b/mkdocs.yml index 336acadb..24ba53a0 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -31,3 +31,4 @@ nav: - Deployment and Release: deploy/deployment-and-release.md - NERSC Spin Runbook: deploy/nersc-spin-runbook.md - HPC API Token Authentication: deploy/hpc-api-token-authentication.md + - Read-Only Database Access: deploy/read-only-database-access.md