Deep Crawler not working

[rimonhanna]

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

1. Go to 'deep crawler'
2. Enter domain "tekunda.com"
3. Click on 'Crawl'
4. See error
   Failed to process https://tekunda.com/: Blocked/Rate Limited (1015) by server for https://tekunda.com/. Try lowering concurrency

I tried to lower concurrency even to 1, IP white listed in cloudflare, nothing.

Expected behavior
It should start crawling

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: MacOS 26.3
• Version 0.3.7

Additional context
Add any other context about the problem here.

[mascanho]

Describe the bug A clear and concise description of what the bug is.

To Reproduce Steps to reproduce the behavior:

1. Go to 'deep crawler'
2. Enter domain "tekunda.com"
3. Click on 'Crawl'
4. See error
   Failed to process https://tekunda.com/: Blocked/Rate Limited (1015) by server for https://tekunda.com/. Try lowering concurrency

I tried to lower concurrency even to 1, IP white listed in cloudflare, nothing.

Expected behavior It should start crawling

Screenshots If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: MacOS 26.3
• Version 0.3.7

Additional context Add any other context about the problem here.

Have you tried whitelisting a user agent in Cloudflare and then defining it in RustySEO as a sole UA?
If it still fails I need to do some digging.
I have no cloudflare accounts that I can test this on to debug it and improve at the moment.
I'll have to check further.

[rimonhanna]

I'm using cloudflare free tier, i tried to whitelist the user agent and the ip address on cloudflare.

[mascanho]

I'm using cloudflare free tier, i tried to whitelist the user agent and the ip address on cloudflare.

After you have done this you need to then specify the user agent in RustysEO.
Or else it will cycle thorugh the predefined ones, and if any of them are blocked it won't go through.

That could be an option.