New Predict Occultation Pipeline using Slurm#1189
Open
glaubervila wants to merge 14 commits into
Open
Conversation
Implements a new prediction occultation module, including task management and worker processes for preparing, submitting, and ingesting results. Introduces new models and management commands to support task lifecycle management and integrates with a cluster job submission system. Enhances the logging system for better traceability with colored output and structured loggers for different workers, facilitating easier debugging and monitoring. Relates to the overall improvement of the backend processing for asteroid prediction tasks.
Introduces new API endpoints for managing prediction tasks and attempts, enhancing the functionality of the application. Registers the PredictionTaskViewSet and PredictionAttemptViewSet in the URL router. Implements serializers for data handling and provides detail retrieval for prediction tasks along with related attempts. Removes unused test and view files to streamline the codebase.
Implements a new feature that groups prediction tasks by their state, enhancing data visualization for users. Introduces a summary component that displays the count of tasks in each state using state chips. Updates the routing to include a dedicated page for prediction tasks, improving navigation. Enhances the API with endpoints for fetching grouped task data, promoting efficient data handling.
Implements a new WorkersHeartbeat model to track the status and uptime of workers. Registers the WorkersHeartbeat API endpoint and integrates it into the admin panel for monitoring. Includes frontend components to display worker status and heartbeat information within the Prediction Tasks interface. Enhances overall monitoring capabilities of the backend processing workflow.
Implements a new LocalExecutor class to facilitate local job submission for testing purposes, improving flexibility in execution environments.
Introduces the BspAsteroid model to manage asteroid data with additional metadata. Implements a management command to update the BspAsteroid table with information from existing files, ensuring the integrity of the asteroid data. Enhances the administrative interface for BspAsteroid, allowing for easier management and search capabilities.
Implements a migration to modify fields in the BspAsteroid model, enhancing database structure with better indexing and help text for clarity. Introduces a new AsteroidEphemerisDAO class to facilitate asteroid ephemeris data retrieval by name. Updates PrepareWorker to integrate the new AsteroidEphemerisDAO for improved asteroid ephemeris information handling.
Eliminates the PredictionAttempt model and its associated views, serializers, and registration in the admin interface. This change streamlines the codebase by removing unused components, simplifying the prediction process, and enhancing maintainability. Additionally, it updates routing and migration files to reflect this removal, ensuring the application remains functional without the deleted elements.
Enhances the enqueue_predictions management command to support asteroid-specific parameters, including start and end dates, asteroid name, and dynamic classes.
Introduces a new 'debug' field to the PredictionTask model to control the cleanup of working directories. Additionally, updates task scripts to log the output manifest and manage task cleanup based on debug status.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.