Skip to content

Feature/job reliability#54

Merged
rosemcc merged 52 commits into
mainfrom
feature/job-reliability
May 6, 2026
Merged

Feature/job reliability#54
rosemcc merged 52 commits into
mainfrom
feature/job-reliability

Conversation

@rosemcc
Copy link
Copy Markdown
Contributor

@rosemcc rosemcc commented May 5, 2026

This pull request introduces enhancements and refactoring to support ActiveScale integration and improve archive job tracking. Key changes:

  • ActiveScale integration
  • Revised archive submission/job tracking model
  • Manifest model and related code have been removed

ActiveScale Integration

  • Added S3/ActiveScale integration module for handling initialisation of Activescale S3 client, as well as generic S3 operations.
  • Added a shell script scripts/check_activescale.sh for verifying ActiveScale connectivity and permissions, including environment loading and proxy handling.
  • Added boto3 and types-boto3-s3 as dependencies for ActiveScale S3 API access in pyproject.toml.

Configuration and Environment

  • New environment variables for ActiveScale configuration (ACTIVESCALE_HOSTNAME, ACTIVESCALE_REGION, etc.) in .env.development and .env.production.
  • Extended the Settings class in src/config.py to include ActiveScale configuration and secrets, using SecretStr for sensitive values.

Archive Submission Model Refactor

  • Refactored the ArchiveSubmission model to track job lifecycle stages using a new JobStage enum, with explicit timestamps and retry/cleanup tracking. Removed legacy fields and manifest relationship.
  • Updated the SubmissionResponse model to reflect the new job stage and timestamp fields, and removed obsolete fields.
  • Updated the frontend api client and functionality to work with the new submission model.

Manifest Model Removal

  • Removed the Manifest model and related code, since for now there's no reason to store the list of files in the db

rosemcc added 30 commits March 11, 2026 11:41
@rosemcc rosemcc merged commit 152b502 into main May 6, 2026
1 check passed
@rosemcc rosemcc deleted the feature/job-reliability branch May 6, 2026 23:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant