Improve Cylc Support

[edoyango]

The CylcProfilingManager class has fallen behind PayuProfilingClass and the below methods need to be implemented:

• delete and archive
• look into running non interactively.

[edoyango]

after executing rose suite-run in the rose dir, a new dir is created /scratch/$PROJECT/$USER/cylc-run/<rose id>.
Copies from the rose dir into the new dir:

app            meta             suite-graph
bin            opt              suite-macros.rc
etc            rose-suite.conf  suite.rc
Jinja2Filters  rose-suite.info  suite-runtime
lib            site             suite-setup.rc

new stuff:

• log => log.<timestamp>
  • log/job/1/* (I think "1" corresponds to the cycle no.) has the following for each step in the pipeline (stored as subdirectories)
    • job: the job script submitted
    • job.err: job's stderr
    • job.out: job's stdout
    • job.status: info about the job's submit, start, and end time and whether the job succeeded or fail. Also include process ID and job ID.
    • job-activity.log: I think this is cylc logging. Less human-readable than the other files.
  • rose-conf/<timestamp>-run.{conf,version}
    • <timestamp>-run.conf is the rose-suite.conf from the rose that was run
    • <timestamp>-run.version is the version of the suite that was run (i.e. reports the repo UUID, the svn revision, last change etc.)
  • rose-suite-run.conf -> rose-conf/20260421T081315-run.conf i.e. a symlink to one of the logs stored in above
  • rose-suite-run.version -> rose-conf/20260421T081315-run.version i.e. a symlink to one of the versions above
  • rose-suite-run.locs: Not entirely sure what this is for it just reports root-dir=/scratch/$PROJECT/$USER
  • suite/log -> log.<timestamp>: the cylc log e.g. task/job polling and health checks etc.
  • suiterc/<timestamp>-run.rc: not too sure what this is for. looks like it's automatically generated from some user defined .rcs? I think it's the graph of the tasks to run and metadata about the tasks etc. Not very human readable.
• .service/ has db passphrase source ssl.cert ssl.pem - probably not relevant to us. Am not sure what it's for.
• cylc-suite.db an sqlite3 db of all the tasks/jobs status/runtime/cycle etc.
• .rose-config_processors-file.db: a db that tracks file dependencies and paths.
• suite.rc.processed: suite.rc with all the jinja templates processed
• work/<cycle>/<task>/: where all the jobs run and store results.
  • each task has a rose-app-run.conf file and whatever outputs was produced by the run.