Scraping ajax pages using python

Question

I've already seen this question about scraping ajax, but python isn't mentioned there. I considered using scrapy, i believe they have some docs on that subject, but as you can see the website is down. So i don't know what to do. I want to do the following:

I only have one url, example.com you go from page to page by clicking submit, the url doesn't change since they're using ajax to display the content. I want to scrape the content of each page, how to do it?

Lets say that i want to scrape only the numbers, is there anything other than scrapy that would do it? If not, would you give me a snippet on how to do it, just because their website is down so i can't reach the docs.

Answer 1

First of all, scrapy docs are available at https://scrapy.readthedocs.org/en/latest/.

Speaking about handling ajax while web scraping. Basically, the idea is rather simple:

• open browser developer tools, network tab
• go to the target site
• click submit button and see what XHR request is going to the server
• simulate this XHR request in your spider

Also see:

• Can scrapy be used to scrape dynamic content from websites that are using AJAX?
• Pagination using scrapy

Hope that helps.