Request get slow with enough data

[Vain05]

I don't want to be nitpicking but when you start having a lot of pages for stuff Like "Top Paths" or others the request can get long for exemple
in my top paths if i want to go to next or prev i need to wait around 6 seconds, for filter word like well "word" it's 2 seconds.

If you want to use this tool for security analysis well it will need a lot of data and i don't know if you can handle it.

(PS: i called it nitpicking because i have around 5 millions accesses)

[BlessedRebuS]

I think there's a tradeoff here: the more data we cache, the heavier Krawl becomes.
At 3M requests, we're running pretty low on memory. That said, we could try to cache more things to see how memory spikes, @Lore09, what do you think?

Also, @Nicolas-Lespinasse, how much RAM does your Krawl instance use right now?

[Vain05]

the more data we cache, the heavier Krawl becomes.

Can't you make the request smarter then ? i mean the pages have a size limit so can't you grab the current page and see mmh i need to grab for id 50-100 or something for page x ?

As for ram i don't know i tried adding monitoring but i can't get the ram because it is in a alpine lxc in proxmox.

[Lore09]

@Nicolas-Lespinasse I'm going to try to implement a more aggressive caching on redis and possibly compute long queries before the dashboard request. That may improve the overall panel speed, but I don't know how to speed up the searchbar queries since they need to fetch all the db to find answers.
Gonna update you soon 🤕

[Lore09]

@Nicolas-Lespinasse I just implemented the stuff I mentioned in the previous comment and deployed it in the demo instance, it looks faster to me.
Could you try to update to the latest dev image? The tag is 2.0.5-dev, you should be able to just change that and redeploy.
Let us know if it improves something! 🪳

[Vain05]

So after checking on the dashboard in now only see 1627 path in 326 pages (before it had the 5millions et thousand of pages , don't know chere it's gone)

Switching in Top pages now take 4s (6 before update)

but my top user-agents which has only 10 pages and 46 total takes 1.5-2s to switch (strange)

the other options are almost instant. and the filter by hand did not change and is still a 2s

[Lore09]

You see less paths because I've introduced a new configuration that can set a lower limit in the "top x thing" queries, since it's a little useless to compute the count of countless paths, user agents etc. that only appear once.
You can set it as you like changing the parameter in the config.yaml (or via env)

Krawl/config.yaml

Line 74 in c5890d0

top_n_min_count: 5

Btw I deployed it in the demo instance and it takes a lot less to switch panels than you (you can check on your own https://demo.krawlme.com/das_dashboard#overview), so I don't understand why you have such poor performances.
Also the filter by searchbar was not updated, since I should guess what anyone wants to search to preload it in cache and I don't really know some ways to optimize this.
I'll keep looking at something to improve it further, let me know if you have any suggestion!

[BlessedRebuS]

@Vain05 any updates on this?

[Vain05]

even with the updates it's still pretty slow but i don't how to help with that