python - Trying to create a web scraper, but the code below keeps printing all html

Question

So im trying to pull my stats from medium.com and have already built a bot to log in, when i get to the stats page and try to print the title, it keeps throwing all the html at me. The print function is there to make sure it's printing the right stuff before I move on:

url = driver.page_source
headers = {"Accept-Language": "en-US, en;q=0.5"}
results = requests.get(url, headers=headers)

soup = BeautifulSoup(url, "lxml")

story_title = []
publication = []
views = []
reads = []
read_ratio = []
fans = []

stats_div = soup.find_all('tr', class_='sortableTable-row js-statsTableRow')
for container in stats_div:
    name = container.td.a.text.find('span', class_='sortableTable-title u-maxWidth450')
    story_title.append(name)

print(story_title)

question from:https://stackoverflow.com/questions/65830087/trying-to-create-a-web-scraper-but-the-code-below-keeps-printing-all-html

Answer 1

Nevermind, got it! Selenium didn't like url = driver.page_source so I just used the link instead.