pandas - Scraping onclick tables in Python

Question

I am trying to scrape Steams hard- and software survey for December 2020 (the table at the bottom of the page). The table is expandable by clicking on one of the parents (e.g. "OS Version"). My goal is to access the tables within these parents.

https://store.steampowered.com/hwsurvey#main_stats

So far I have tried to retrieve this information with requests and BeautifulSoup (with different parsers) but Beautifulsoup always returns an TypeError: 'NoneType' object is not callable. After searching unsuccessfully for an API, I tried Selenium in combination with pd.read_html(). With this approach I was able to access at least the y-labels from the charts above the table but not the desired table below:

import pandas as pd
from selenium import webdriver

url = "https://store.steampowered.com/hwsurvey#main_stats"
opt = webdriver.FirefoxOptions()
opt.add_argument('-headless')
driver = webdriver.Firefox(options=opt)
driver.get(url)

pd.read_html(driver.page_source)

I am appreciative of any suggestion that may help me to overcome this problem.

question from:https://stackoverflow.com/questions/65835743/scraping-onclick-tables-in-python

Answer 1

You can try again from this code:

import requests
from bs4 import BeautifulSoup

url="https://store.steampowered.com/hwsurvey#main_stats"
headers = {"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6),AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36"}

response = requests.get(url, headers=headers).text
soup =  BeautifulSoup(response,"html.parser")
names=soup.find_all("div",{"class":"stats_col_left"})
os=soup.find_all("span",{"id":"osversion_val_1_on"})
#val=soup.find_all("div",{"class":"stats_col_mid"})
list_names=list()
for i in names:
    i=i.text
    i=i.strip("xa0 ")
    list_names.append(i)
    list_names = [x for x in list_names if x]