Dynamic Usage in large projects

[theduke]

Has apalis been used in large projects with hundreds of different kinds of tasks?

The current API design seems to make this kind of usage incredibly awkward, with storage backend carrying around the args generic, and needing to manually start workers for each type.

It also seems like a process processing many different tasks would do an incredibly amount of redundant work while waiting for tasks?

I guess a workaround would be to use an enum for the args , but that would lose a lot of observability and control.

It would seem much more sensible to have an underlying dynamic API that can be used in case the typed API is not desirable.

[geofmureithi]

I think the best way to discuss your issue is to include an example use case. Because there are several approaches that are usable.

Eg Could you explain what you mean by this? What storage are you using? What version.

It also seems like a process processing many different tasks would do an incredibly amount of redundant work while waiting for tasks?

Possible solutions even without much context:

1. Enum. If the tasks are related you can use enum + dispatch as an approach eg using enum_dispatch. This can be ideal if you want only one worker and the tasks are related.

2. MakeShared and Generics This includes using a Shared connection and generics to handle multiple tasks using the same connection and leveraging rust's generics. Eg in this example

async fn send_reminder<T: DoSomeWork>(
        task: T,
        wrk: WorkerContext,
    ) -> Result<(), BoxDynError> {
        task.do_some_work().await
        Ok(())
}

let int_worker = WorkerBuilder::new("worker-2")
        .backend(int_store)
        .build(send_reminder);
let map_worker = WorkerBuilder::new("worker-1")
        .backend(map_store)
        .build(send_reminder);
tokio::try_join!(int_worker.run(), map_worker.run()).unwrap();

3. MakeShared and functional approach
   This includes creating a function that outputs a worker:

use futures::future::join_all;

let mut store = SharedPostgresStorage::new(pool);

fn build_worker<S: Service<PgTask<T>, T>(store: &mut SharedPostgresStorage, svc: S) -> BoxFuture {
   let backend = store.make_shared().unwrap();
   WorkerBuilder::new("worker-1")
        .backend(backend)
        .build(svc)
        .run()
        .boxed()
}

let workers = Vec::new();
for task in tasks {
     workers.push(build_worker(&mut store, task));
}

join_all(workers).await

4. Macros: There is still the possibility of using macros! to even make this process easier.

What does this mean?

apalis looks to not be opinionated in this because each approach would not be ideal in most cases. The problem you are experiencing is more about how you structure your rust code rather that an API that apalis can provide. The MakeShared trait allows you to reuse the connection and polling logic meaning that this would be super efficient and not overload the system in a scenario you have hundreds of jobs.

What next?

I think it will be wise to include concrete examples of these approaches to offer better understanding to this question as it has been raised several times and usually comes back to these three options. If you feel that this does not address your issue effectively I am willing to keep this conversation going :).