[DevOps]: Optimize DB-backed ingestion state queries for large performance archives

[tomvothecoder]

Summary

Coordinate this work with #261.

PRs #260 and #272 reduced repeated archive processing, but the ingestor still requests the full history for a machine. As that history grows, the request may become slower and larger than needed.

Goal

Use the results of the incremental scan from #261 to request only the ingestion state needed for new or changed archive content.

Coordination with #261

• [DevOps]: Add check-point based scanning to optimize ingestor discover phase #261 should be implemented first and identify new or changed archive paths.
• This issue should use those paths to limit the state requested from SimBoard.
• Keep the implementation work separate so scanning and state-query changes can be tested and reviewed independently.
• Coordinate the shared information passed between the two steps.

Acceptance criteria

• Normal incremental runs request only relevant ingestion state.
• Full rescans and checkpoint resets remain correct.
• Existing ingestors remain supported during the transition.
• Measure request size and runtime before and after the change.
• Document how this issue works with [DevOps]: Add check-point based scanning to optimize ingestor discover phase #261.

Scope

This issue covers database query and response efficiency. Filesystem traversal and checkpoints are covered by #261.

[TonyB9000]

@tomvothecoder I note here that the function _scan_archive called in hpc_upload_archive_ingestor.py (imported from nersc_archive_ingestor.py) is passed in "state", and consists of these three sub-functions:

    discovery_stats = _new_discovery_stats()
    grouped_executions = _discover_case_executions(
        config.archive_root, metadata_locator=metadata_locator, stats=discovery_stats
    )
    scan_results = _build_case_scan_results(grouped_executions)

    candidates = _build_ingestion_candidates(
        scan_results, state, max_cases_per_run=config.max_cases_per_run
    )

Critically, the entire archive is passed through _discover_case_executions(, and through _build_case_scan_results, before we apply "state" - ostensibly to eliminate case/executions previously completed.

In order to address efficiency here, it is not enough to filter the state request to (say) "entries_since" (date). The state must be applied in the function _discover_case_executions to gain real advantage. Otherwise, we unnecessarily collect and process most all "grouped_executions".

If "state" consists essentially of (case_id/execution_id) pairs, this should be easy. But if state represents a great deal of subordinate detail about each pair ("missing optional files", for instance) then much more work may be needed. From my perspective, any "completed" (case/exec) is just that - completed - an no further consideration is necessary.

[TonyB9000]

@tomvothecoder Continuing the above commentary:

def _discover_case_executions(...):
    ...
    for dirpath, dirnames, _ in os.walk(archive_root):
        for dirname in dirnames:
            if not EXECUTION_DIR_PATTERN.fullmatch(dirname):
                continue
            if stats is not None:
                stats["execution_dirs_scanned"] += 1

            case_dir = Path(dirpath)
            if not _validate_execution_dir(
                case_dir,
                dirname,
                metadata_locator=metadata_locator,
                stats=stats,
                skip_log_state=skip_log_state,
            ):
                continue

            if stats is not None:
                stats["execution_dirs_accepted"] += 1
            grouped.setdefault(str(case_dir.resolve()), set()).add(dirname)

In the above code, at the very point we issue case_dir = Path(dirpath), we have BOTH {case_dir + dirname), where dirname matches an exec_id pattern. HERE is where we could circumvent a great deal of unnecessary processing, if "state" sufficed to tell us whether this (case_id,exec_id) pair has already been ingested as "complete". I assume it cannot become "more complete" upon further examination.

We can avoid "_validate_execution_dir", and the dirname need not be added to the groups that are subsequently processed by _build_case_scan_results(grouped_executions).

[tomvothecoder]

I think this is is related to #261