database model structure for cloud compute ORC-OS nodes

[hcwinsemius]

• investigate other examples (e.g. WebODM)
• implement a logical model with logical fields like url, JWT, description
• show statusses on the admin front end (investigate new async options for background pinging as option).

[hcwinsemius]

Here we investigate WebODM as an example of how task distribution is done. This is because WebODM is quite comparable to our situation.

WebODM example

Similarities with LiveORC in terms of requirements:

1. Worker nodes (which can be extended/added) which are entirely separate from the central API.
2. Works with quite large datasets (photos/videos)
3. Needs to report back with some callback mechanism after processing.
4. Requires some sort of state handling to monitor state of jobs.

WebODM consists of

• Django web back and front end (+admin)
• PostgreSQL database
• task distribution via Celery
• In docker, a redis message broker
• several tasks that are distributed via celery

Important thing to note: WebODM communicates with nodes via normal API https requests. it never communicates the other way around. So remote nodes will never hold or store keys to WebODM. This ensures that WebODM is very secure, even if a processing node gets exposed.

Processing nodes

Processing nodes can be added/configured. The information about the physical processing node is stored in a data model that has details such as the host/port name, token, a readible name, api version, last refresh, and certain options and limitations.

Important methodes of these nodes include:

• 'update_node_info' method that checks the last status of a node. This is also used to update the last time it was refreshed and whether the node seems to be online or not.
• process_new_task which actually sends a task including all the data (images and videos) to the api client side where it is then truly processed. the client then returns a task uuid after which the node instance can wait until the task is done. It also has a progress_callback which can be used to provide a means to return updates of the task intermittantly over an http request.
• a number of methods that cancel, delete, download, restart, tasks after they have started.

In summary, once the celery worker has identified which processing node to send a task to, all communication to an individual node is handled by the ProcessingNode instance.

API client

A separate API client is written called PyODM which handles requests to the nodes at a reasonably high level.
Most important parts are that it handles multipart uploads. Likely this is not required as we usually deal with just one single file which can be just opened and then passed as a file input argument. Roughly what happens:

• first a new task is initialized with all settings and run options, a uuid is usually made by the physical node.
• then files get uploaded to a task with a shared uuid (this is happening mith queue management so that uploads progress in the background with a percentage number for updating to the front end)
• then the task is committed using that same uuid to identify which uploads belong to which task. This returns a Task instance, which then further down is used to update the task in the WebODM database.

Task object

Each task is also stored in the database and can be updated. The UUID is the primary key. Once the task is retrieved, it can be used to check what assets are available, what the status is, and use methods to download or update the task. The most important method for this project is process as the logic for processing is in here.
Most importantly, process searches for a processing node that is not doing much or has little in the queue and submits a job, including upload callback to that. the callback arranges that the task itself gets updated with progress information. This is an interesting approach to enable a user to see how far a job is.

After process is called, the task gets a uuid and a processing node oin which the processing is happening. During process eventually after the processing is done, the attached ProcessingNode instance will be called and assets are downloaded. Finally the model instance is saved to show status complete, and ensure that the user sees updated.

Concurrency

After studying the celery worker structure, we conclude that a celery worker is started that can handle mutiple concurrent tasks using the amount of processors as a guidance on how many. process in particular is a long running task, especially if things get queued because there are not enough nodes available to process and tasks get queued. We imagine that on the node side, queuing happens with concurrency=1 so that only one task is executed at the same time, and that a api call to check on the amount of queued tasks must be available to check which node is the most free. On the LiveORC side, the concurrency will likely not be much of a problem