[DevOps]: Add dry-run modes for comprehensive collection and ingestion testing

[tomvothecoder]

Task Details

Need to fill out in detail.

Current notes:
Run 2 – creates tar to see it works
Run 3 – does not update database, just checks if receives

Steps to Complete

No response

Additional Notes

No response

[TonyB9000]

@tomvothecoder Will we have a "DRY_RUN_1", "DRY_RUN_2" and "DRY_RUN_3" to enable these testing levels? When can I proceed, and how can I contribute to move things along here?

[tomvothecoder]

@tomvothecoder Will we have a "DRY_RUN_1", "DRY_RUN_2" and "DRY_RUN_3" to enable these testing levels? When can I proceed, and how can I contribute to move things along here?

If you can provide requirements for each mode that'd help. I can assign the tasks to an AI agent for implementation.

[tomvothecoder]

I don't think we actually need multiple dry-run modes because it adds code implementation overhead and it becomes less useful once the infrastructure is stable.

If our goal is to perform additional validation, we can simply do DRY_RUN=false with MAX_CASES_PER_RUN=1. This is completely okay to do on the development backend and database.

Codex suggests that we don't need to add dry-runs. Instead, we can use these existing controls for validation.

We do not need DRY_RUN_1 / _2 / _3. Existing controls cover validation safely:

• DRY_RUN=true: scan and report candidates only; no API calls or persistence.
• DRY_RUN=false with MAX_CASES_PER_RUN=1: small, real ingestion validation.
• DRY_RUN=false without cap: normal scheduled ingestion.

Keep DRY_RUN permanently for onboarding, deploy/config checks, archive-scope changes, and diagnosing operational issues. Use one-off debug scripts only for exceptional data-specific investigations; do not turn them into permanent runner modes unless they become recurring operational needs.

[tomvothecoder]

Current notes:
Run 2 – creates tar to see it works
Run 3 – does not update database, just checks if receives

When you do DRY_RUN=false with MAX_CASES_PER_RUN=1, I can check the database and SimBoard backend logs. We can schedule a paired-programming call to do this live if it helps you get immediate feedback.

[TonyB9000]

Your recent work in augmenting collection to update finer state (what is or is-not missing in given execution directories, or which "incomplete" execution directories have been seen before) complicates perhaps - latter dry-run operations.

Am I correct in the assumption that, beyond state-retrieval, no further HTTP communication with the backend occurs until the packaged updates are finally pushed? That is, the discovery of new-but-incomplete exec directories is not shared with teh NERSC backend until the successful candidates are submitted?

It makes a huge difference if the flow looks like

    For each case found
        discover compete stuff
        package complete stuff
        send complete stuff

compared to

    For each case found
        discover complete stuff

    For all discovered complete stuff
        package completed stuff

    For all completed packages
        transfer packages

For the latter structure, it is trivial to put a "premature exit" between the major phases for dry-run support. With the former structure, one must not exit, but skip further processing and loop-back to continue (discovery, packaging, etc).

[tomvothecoder]

Thanks, Tony. I've summarized everything below.

Your batch-phase observation is correct. Current non-dry-run flow is:

1. Fetch state/checkpoints.
2. Scan all eligible content.
3. Persist newly discovered immutable outcomes: accepted, rejected incomplete, and rejected invalid.
4. Submit selected cases.
5. Persist checkpoints.

My current view

I believe existing options are sufficient for now:

1. DRY_RUN=true exits after discovery: it validates archive access/content without HTTP, tar creation, or writes.
2. Development backend/database plus DRY_RUN=false MAX_CASES_PER_RUN=1 exercises the full workflow with limited submission scope: discovery, packaging, HTTP, backend handling, persistence, and logs.

This end-to-end test should help identify problems at each stage without adding new modes now. It writes discovery results for newly found executions across the scan, so it should use a development database where test records are acceptable or can be reset afterward.

More isolated modes may be useful later, but they do not appear necessary for current validation.

We already have a dedicated development environment for this test. Once I create a limited-access PostgreSQL account for you, you can inspect development-database records while we run DRY_RUN=false MAX_CASES_PER_RUN=1 end-to-end. We can also do this together on a call.

Possible future additions

These modes could be valuable because they isolate failures that end-to-end ingestion combines. They also require non-trivial implementation and test coverage, especially backend validation without database writes. Do you agree that current validation is sufficient for now, or do you see a larger recurring need for either of these additional workflows?

1. Check HPC tar creation without sending anything

This would create a tar file for one selected HPC case, allow inspection, and stop before HTTP requests or database updates.

• Value: separates package/tar failures from network, API, and database failures.
• Complexity: moderate. Tar creation is currently coupled to upload transport, so packaging must first become a separate runner phase.
• Applies only to HPC upload. NERSC path ingestion does not create tar files.
• If this becomes a repeated onboarding need, a clear mode such as RUN_MODE=scan|package|ingest would be easier to understand than numbered DRY_RUN_1, _2, etc.

2. Check backend request handling without saving data

This would let SimBoard receive and validate a request without database updates.

• Value: separates request/API validation failures from persistence failures.
• Complexity: higher. Runner cannot prevent backend persistence after normal request is received; this needs backend design, authorization decisions, and test coverage.
• For now, our resettable development database may be enough. A mock backend or validation-only endpoint could be added later if this becomes a repeated need.

[TonyB9000]

@tomvothecoder You wrote:
DRY_RUN=true exits after discovery: it validates archive access/content without HTTP, tar creation, or writes.
In truth, it DOES involve HTTP at the start, to fetch the state, correct?

I agree we can exercise (say) tar creation by limiting " MAX_CASES_PER_RUN=1". I was thinking of avoiding having the backend even deal with anything by means of (1). I agree that RUN_MODE=scan|package|ingest is preferable to numbered levels.

You wrote:

Current non-dry-run flow is:

    1. Fetch state/checkpoints.
    2. Scan all eligible content.
    3. Persist newly discovered immutable outcomes: accepted, rejected incomplete, and rejected invalid.
    4. Submit selected cases.
    5. Persist checkpoints.

But this breakdown is hard to map to "scan | package | ingest" (or "scan | package | transmit | ingest "). How does one "persist" prior to "submit", etc?

By my (simple-minded) way of thinking, if we

Complete Scanning

Complete Packages

Complete Transfers

as major phases, then a simple

    If RUN_MODE == PACKAGE:
        # log messages and exit

after "Complete Packages" is sufficient to test just through package (tar) creation. The existing DRY_RUN would be implemented by a similar

    If RUN_MODE == SCAN:
        # log messages and exit

just after "Complete Scanning" and before "Complete Packages".

About "MAX_CASES_PER_RUN = 1": Does this mean "just one completed case"? What if the first case encountered has only "new but incomplete" execs? Does that count as a "CASE" even though no package would be created?

(Sorry for beating this horse to death! :) )

(P.S. I see now from (1), package is coupled to transport. That makes this more difficult.)

[tomvothecoder]

@tomvothecoder You wrote:
DRY_RUN=true exits after discovery: it validates archive access/content without HTTP, tar creation, or writes.
In truth, it DOES involve HTTP at the start, to fetch the state, correct?

Thanks, this is correct on the main branch.

On feature/154-ingestion-sites. I actually pushed Commit 71821a0 — “Allow offline ingestion dry run" which allows dry_run=True without HTTP endpoint/state/checkpoint calls (e.g., when backend is down).

I'm adding the DRY_RUN_USE_REMOTE_STATE env variable with default set to True (commit bfc36a2).

But this breakdown is hard to map to "scan | package | ingest" (or "scan | package | transmit | ingest "). How does one "persist" prior to "submit", etc?

I generated this ingestion diagram using GPT against the SimBoard backend codebase. Let me know if it makes sense.

The current flow persists discovery validation results before submission; it does not persist packages or successful-ingestion state. DRY_RUN exits after scanning, validation, and candidate selection, so it creates no tarballs or sends ingestion requests. A package-only mode would be new behavior because tar creation occurs only per HPC upload attempt; path-based ingestion has no package step.

flowchart TD
    A[Fetch prior state / checkpoints] --> B[Scan filesystem]
    B --> C[Validate executions]
    C --> D{Validation outcome}
    D -->|Incomplete / invalid| E[Record validation outcome in memory]
    D -->|Valid and new| F[Build case candidate]
    D -->|Already processed| G[Skip]
    F --> H[Apply MAX_CASES_PER_RUN]
    E --> I{DRY_RUN?}
    G --> I
    H --> I
    I -->|Yes| J[Log candidates and exit\nNo writes, packages, uploads, or ingestion]
    I -->|No| K[Persist new discovery results]
    K --> L{Runner type}
    L -->|HPC upload| M[For each selected case:\ncreate temporary tar.gz]
    L -->|Path-based| N[For each selected case:\nsubmit path]
    M --> O[Transmit request to API]
    N --> O
    O --> P[API ingests case]
    P --> Q{Success?}
    Q -->|Yes| R[Record processed execution IDs]
    Q -->|No| S[Leave eligible for later run]

Loading

About "MAX_CASES_PER_RUN = 1": Does this mean "just one completed case"? What if the first case encountered has only "new but incomplete" execs? Does that count as a "CASE" even though no package would be created?

With MAX_CASES_PER_RUN=1, runner selects up to one case that has at least one new, valid (complete) execution ready for ingestion.

A case containing only new-but-incomplete executions produces no candidate/package, so it does not consume cap. Runner skips/logs those executions, then can select next eligible case.