python - Handling an infinite scroll UI in BeautifulSoup

Question

I'm looking at how to scrape Linkedin source (https://www.linkedin.com/mynetwork/invite-connect/connections/) but it seems impossible with infinite scroll. How to deal with it? I don't want to use Selenium (want to implement as web service later on).

import bs4
from bs4 import BeautifulSoup
import requests

def scraping(webpage):
    headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'}
    response= requests.get(str(webpage), headers=headers)
    soup = BeautifulSoup(response.text,"html.parser")
    print(soup)

scraping('https://www.linkedin.com/mynetwork/invite-connect/connections')

Answer 1

BeautifulSoup can only help with the HTML you give it; you'll need to cause LinkedIn to return more HTML. The content isn't in the HTML you have, so you must get it. The browser is probably running LinkedIn's javascript to notice that you're scrolling and therefore it needs to fetch more content and inject more HTML in the page - you need to replicate this content fetch somehow.

Bad news: BeautifulSoup isn't aware of APIs or javascript. You'll need another tool.

Good news: there are tools for this! You could certainly use Selenium, that would probably be the simplest way to solve this, since it would replicate the browser environment pretty well for these purposes.

If you are absolutely committed to not using Selenium, I recommend you deep-dive on the LinkedIn site and see if you can figure out which bits of javascript are responsible for fetching more data, and replicate the network requests they make, and then parse that data yourself.

For most people, though, Selenium will be the right answer.