Crawler seems slow

[ashfame]

Is it just me or the crawler seemed slow even with 16 workers?

I imagine it's slow because the browser is rendering the whole page before doing anything with them, rather than just make out stuff with HTML, but thought I would check-in regarding it anyway.

[Veejay]

Hi,

Yeah it's definitely not fast. There are two main reasons for that:

1. I built this as an exercise, mostly for fun with no thought about performance, usability or anything. Its highly likely this doesn't handle all sorts of corner cases I didn't think of.
2. This is not really a crawler in the classical sense because it uses a full Chromium instance under the hood

Feel free to clone this and do what you want with it, bumping the version of Puppeteer should just be a matter of modifying the package.json file.