Skip to content

[ImportTasks]: Split bulk-import into multiple little imports #230

@StevenHosper

Description

@StevenHosper

Currently one entire bro-domain import runs as a single task.
To optimize speed, we should aim to split these into celery tasks, like the uploadtask.

This way, during request wait times (which is 95% for import tasks), we are able to process other objects. Speeding up the time to complete by a landslide.

To convert this import flow into a celery task:

  1. Identify each step
  2. Convert the step to a function
  3. Convert the function to a celery task
  4. Link the celery tasks in a task chain
  5. Correct the bulk import to send celery tasks instead

*Optional: should maybe add a bulk-import and importtask object, instead of the current importtask object. To follow the setup of UploadTasks

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions