[DevOps]: Implement remote data collection job at Chrysalis for ingestion

[tomvothecoder]

Objective

Extend SimBoard metadata ingestion beyond Perlmutter/NERSC, starting with Chrysalis, using a shared scheduler-agnostic Python entrypoint plus thin site-specific wrappers.

Current branch: feature/154-ingestion-sites

The branch currently adds:

• backend/app/scripts/ingestion/hpc_archive_ingestor.py: shared entrypoint that delegates to the existing NERSC archive ingestor for now.
• backend/app/scripts/ingestion/sites/chrysalis.sh: thin Chrysalis Jenkins wrapper.
• README updates documenting the shared ingestor and Chrysalis defaults.
• A test that verifies the generic entrypoint delegates through the existing ingestor path.

Site Priorities

• Chrysalis: priority once available again; Jenkins-based workflow.
• Frontier: apply for account; likely next priority.
• Aurora: apply for account; likely next priority.
• Compy: apply for account.
• Anvil: removed from scope.

References

• PACE overview: https://e3sm.atlassian.net/wiki/spaces/EPG/pages/776437853/Performance+Analytics+for+Computational+Experiments+PACE
• PACE collection/upload reference: https://e3sm.atlassian.net/wiki/spaces/EPG/pages/5477335106/PACE+Collection+and+Upload+Reference
• Existing site script wrappers: https://github.com/E3SM-Project/E3SM_test_scripts/tree/master/jenkins
• Existing PACE archive script: https://github.com/E3SM-Project/E3SM_test_scripts/blob/master/util/pace_archive.sh

[tomvothecoder]

Hey @rljacob, I’m scoping SimBoard metadata ingestion beyond Perlmutter/NERSC.

Your PACE Confluence page lists Chrysalis, Compy, Aurora, Frontier, and Anvil. Should we support all, or prioritize 1-2 first?

My assumption: start with Chrysalis, then reuse same scheduler-agnostic ingestion framework elsewhere.

Current runners from page:

• Jenkins: Chrysalis, Compy, Anvil
• Cron: Frontier
• ALCF GitLab: Aurora

Does priority/order look right, and are these runners still current? Also, I would need to request access to Compy, Frontier, or Aurora so I'm blocked there for now.

[rljacob]

Anvil can be removed. You should apply for accounts on Frontier, Aurora and Compy. The first 2 will probably be fastest. Make Chrysalis the priority once it comes back. Rob From: Tom Vo ***@***.***> Date: Tuesday, April 28, 2026 at 1:21 PM To: E3SM-Project/simboard ***@***.***> Cc: Jacob, Robert L. ***@***.***>; Mention ***@***.***> Subject: Re: [E3SM-Project/simboard] [Enhancement]: Scope automated metadata ingestion at other sites (Issue #154) This Message Is From an External Sender This message came from outside your organization. [https://avatars.githubusercontent.com/u/25624127?s=20&v=4]tomvothecoder left a comment (E3SM-Project/simboard#154)<https://urldefense.us/v3/__https://github.com/E3SM-Project/simboard/issues/154*issuecomment-4338001402__;Iw!!G_uCfscf7eWS!fdyZLsYJBPlI7ti3lBPdE5YdYox-ZsuIlLmJk3if_IE3AvTlz4v1azpwSPH-W_YdQEwN1KOXdnCJv1yvhSHh4lY$> Hey @rljacob<https://urldefense.us/v3/__https://github.com/rljacob__;!!G_uCfscf7eWS!fdyZLsYJBPlI7ti3lBPdE5YdYox-ZsuIlLmJk3if_IE3AvTlz4v1azpwSPH-W_YdQEwN1KOXdnCJv1yvg4g8e0o$>, I’m scoping SimBoard metadata ingestion beyond Perlmutter/NERSC. Your PACE Confluence page<https://urldefense.us/v3/__https://e3sm.atlassian.net/wiki/spaces/EPG/pages/5477335106/PACE*Collection*and*Upload*Reference__;KysrKw!!G_uCfscf7eWS!fdyZLsYJBPlI7ti3lBPdE5YdYox-ZsuIlLmJk3if_IE3AvTlz4v1azpwSPH-W_YdQEwN1KOXdnCJv1yvF0pnqVo$> lists Chrysalis, Compy, Aurora, Frontier, and Anvil. Should we support all, or prioritize 1-2 first? My assumption: start with Chrysalis, then reuse same scheduler-agnostic ingestion framework elsewhere. Current runners from page: * Jenkins: Chrysalis, Compy, Anvil * Cron: Frontier * ALCF GitLab: Aurora Does priority/order look right, and are these runners still current? Also, I would need to request access to Compy, Frontier, or Aurora so I'm blocked there for now. — Reply to this email directly, view it on GitHub<https://urldefense.us/v3/__https://github.com/E3SM-Project/simboard/issues/154*issuecomment-4338001402__;Iw!!G_uCfscf7eWS!fdyZLsYJBPlI7ti3lBPdE5YdYox-ZsuIlLmJk3if_IE3AvTlz4v1azpwSPH-W_YdQEwN1KOXdnCJv1yvhSHh4lY$>, or unsubscribe<https://urldefense.us/v3/__https://github.com/notifications/unsubscribe-auth/AAHHG7AW33RWP2RPJSXA2QL4YDZBHAVCNFSM6AAAAACW6BWFDCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DGMZYGAYDCNBQGI__;!!G_uCfscf7eWS!fdyZLsYJBPlI7ti3lBPdE5YdYox-ZsuIlLmJk3if_IE3AvTlz4v1azpwSPH-W_YdQEwN1KOXdnCJv1yv26dcKnY$>. You are receiving this because you were mentioned.

[TonyB9000]

I would like to acknowledge @rljacob for his response to my question, to wit:

Tom Vo suggested you could point me to someone about getting "the e3smtest account for running jobs" (this would be recurring daily performance archive-ingestion jobs on Chrysalis, likely involving "Jenkins", pushing data to "SimBoard" on NERSC.  (If that makes it sound as if I know what I'm talking about - don't be fooled).  Any clues appreciated!

with

To get a jenkins job running on Chrysalis, here’s what you need to do:

You’ll need to make a PR to https://github.com/E3SM-Project/E3SM_test_scripts/
E3SM-Project/E3SM_test_scripts
Test scripts for cron and jenkins jobs

The job is split into two scripts. Tom wants Simboard to do something similar to PACE.

The pace jenkins script is here: https://github.com/E3SM-Project/E3SM_test_scripts/blob/master/jenkins/chrysalis_pace.sh

So should first create chrysalis_simboard.sh script that is similar and add it to that directory.

That script basically checks out the repo and calls another script. https://github.com/E3SM-Project/E3SM_test_scripts/blob/master/util/pace_archive.sh

So you should also make a simboard_ingest script and put it in the util dir.

This is done so that if we want to change what happens, we can just edit the script in util and not ask to have the jenkins updated.
You can run the util script yourself on chrysalis for testing. Once its working, get it hooked up to jenkins.

When you’re ready to start the jenkins, go to https://e3sm.atlassian.net/servicedesk/customer/portal/2 and fill out a “Need Something Else” request to explain to Wade what you want.

And continues with:

That script basically checks out the repo and calls another script.   https://github.com/E3SM-Project/E3SM_test_scripts/blob/master/util/pace_archive.sh

so you should also make a simboard_ingest script and put it in the util dir.
this is done so that if we want to change what happens, we can just edit the script in util and not ask to have the jenkins updated.
you can run the util script yourself on chrysalis for testing.  Once its working, get it hooked up to jenkins.

When you’re ready to start the jenkins, go to https://e3sm.atlassian.net/servicedesk/customer/portal/2 and fill out a “Need Something Else” request to explain to Wade what you want.

Thanks again Rob! (And thank you @tomvothecoder for a kick in this direction. I need to become habituated to using github for relevant conversation tracking.)