web scraping - Python Webscraping not return the right text, and sometimes no text at all

Question

I'm trying to retrieve the price of the item from this amazon page, URL:

 https://www.amazon.com/FANMATS-University-Longhorns-Chrome-Emblem/dp/B00EPDLL6U/

Source Code

from bs4 import BeautifulSoup
import requests

text = "https://www.amazon.com/FANMATS-University-Longhorns-Chrome-Emblem/dp/B00EPDLL6U/"
page = requests.get(text)

data = page.text

soup = BeautifulSoup(data, 'lxml')

web_text = soup.find_all('div')

print(web_text)

---

Everytime I run the program, I get an output of html that's nothing similar to that of the webpage, saying things like:

" Sorry! Something went wrong on our end. Please go back and try again..."

I'm not sure what I'm doing wrong, any help would be much appreciated. I'm new to python and webscraping so I'm sorry if my issue is super obvious. Thanks! :)

question from:https://stackoverflow.com/questions/65912664/python-webscraping-not-return-the-right-text-and-sometimes-no-text-at-all

Answer 1

Website is serving content dynamically what request could not handle, use selenium instead:

from selenium import webdriver
from bs4 import BeautifulSoup

url = 'https://www.amazon.com/FANMATS-University-Longhorns-Chrome-Emblem/dp/B00EPDLL6U/'
driver = webdriver.Chrome('C:Program FilesChromeDriverchromedriver.exe')
driver.get(url)
time.sleep(3)

html = driver.page_source
soup = BeautifulSoup(html,'html.parser')
print(soup.select_one('span#priceblock_ourprice').get_text())
driver.close()