Have I understood follow correctly?

[dynabler]

After my issue with .split()[] #32 and coincidently the release of YouTube video 'Introduction to Flyscrape', I have come to the realization that I used follow, depth, allowedURLs/blockedURLs differently than Flyscrape intends to.

Flyscrape follow

As evident in documentation and shown in YouTube video 'Introduction to Flyscrape', the final page being scraped is a list page.

flowchart LR;
start-url --> list-page-1
start-url --> list-page-2
start-url --> list-page-3

Loading

How I was using it

What I was doing is using URL pagination #38 + follow, so the final page being scraped is a single page

flowchart LR;
start-url --> list-page-1-->single-pages-11[single page 1]
list-page-1-->single-pages-12[single page 2]
list-page-1-->single-pages-13[single page 3]
start-url --> list-page-2-->single-pages-21[single page 1]
list-page-2-->single-pages-22[single page 2]
list-page-2-->single-pages-23[single page 3]

Loading

My question is, have I understood it correctly? Am I missing something? Without hack #38, Flyscrape can only go one step, from url to list-page?

[Davasny]

Hey! In my case I need to go from "list-page-1" into "list-page-1-1" (subpage) and then into "single page n" where links are not possible to predict like in #38. Do you know if there is any way to do it using flyscrape?

[dynabler]

It is possible, but very difficult to wrap your head around it. It's not just Flyscrape, but any scraper has trouble following links. To be precise, scrapers can follow links, but the amount of empty data you end up with requires cleaning-up, making it not worth the effort. If you really want to follow links, as you describe: use a combination of follow, allowedURLs, blockedURLs and depth to control how Flyscrape moves through pages. Expect to spend hours getting it right. Maybe the following suggestion are better options.

A better option is: reconsider where you're going to start your "start-url". So skip list-page-1 and start at list-page-1-1/2/3 etc. Can you do that? Let me know.

Other stuff to check: https://example.com/robots.txt and sitemap.xml, maybe there's a list of URLs you can get, skipping list pages (or parts of it) entirely. Let me know if there's a sitemap.xml (you can scrape sitemap.xml and use the URLs to single pages to scrape those directly.

Your comment: then into “single page n” where links are not possible to predict like in #38. Which links are not predictable? Those on list-page-1-1? Or single page? If it's single page, it's okay, since you don't need to follow links after single page, that's the final destination aka data page. If links on list-page-1-1 is not predicable, check if it gives some sort of Fetch/XHR (Chrome DevTools --> Network --> select Fetch/XHR). It may be just a JavaScript that fetches a JSON, which you can get directly.