Hi,
First of all, thanks a lot for this contribution–super cool.
I was wondering if you were planning to add batch inference? It seems like the memory consumption when running prediction is not so large, I wonder if running inference in batches might speed up the process quite a bit.
Happy to hear your thoughts!
Hi,
First of all, thanks a lot for this contribution–super cool.
I was wondering if you were planning to add batch inference? It seems like the memory consumption when running prediction is not so large, I wonder if running inference in batches might speed up the process quite a bit.
Happy to hear your thoughts!