scrapping is for static site update it for dynamic site

[anant-dev]

[ashwini0529]

Dynamic as in what?
Describe?

[ankitjain28may]

Is it working at all because i guess flipkart's data is all dynamic?

[anant-dev]

@ashwini0529 Dynamic in the sense that flip-kart is using React js for final rendering of the website on client side so the data i am fetching is on request which gets modified by browser and shown to us so i have no use of initial html code that is saved.
You can see this by comparing the page source code and inspect code of the list of products you will find the two different.

[anant-dev]

@ankitjain28may it was working but as they have switched to react its not working that is why i have opened this issue

[ankitjain28may]

Ya thats the problem I recently faced while scrapping snapdeal n flipkart, you can use selenium to scrape the data..!

[anant-dev]

@ankitjain28may Selenium is a web automation tool that creates an instance of browser each time you run the script which is not applicable for server side scripting as it needs a machine with browser to perform the operation. I have searched a few material and got to know that it can be done using scrapy only if you know the Ajax calls and can be easily performed by various js frameworks like casperjs.

I will try to do this using scrapy.
thanks for you suggestion .

[ankitjain28may]

@anant-dev it can be easily implemented through selenium I have recently used it for snapdeal... N its best for automation of your tasks too.. But scrapy is actually made to do scrapping so if you are going to do it from scratch than go for selenium and bs4 else it can be done using scrapy..!

[anant-dev]

@ankitjain28may I am not doing it from scratch and as i told you for running script on server side we cannot use selenium, i have used selenium as well as scrapy but for this purpose as i mentioned i ll go with scrapy.
But, thanks for your suggestion