json - BeautifulSoup <small> tag

Question

I want to scrape the text between <> element. HTML code where I want to scrape "text" inside small (which is was: 27.00). The HTML is <> class="product-views-price-old" Was: ￡27.00 <> My code is:

from bs4 import BeautifulSoup
import requests
url = "https://www.petshop.co.uk/Dog"
r = requests.get(url)
soup = BeautifulSoup(r.content)
for old_price in soup.find_all("small", class_ = "product-views-price-old"):
    print(old_price)

The above code gives me nothing. Even no error. How can I scrape the text between <> tags?

question from:https://stackoverflow.com/questions/65888966/beautifulsoup-small-tag

Answer 1

Content is served dynamically, so you wont get it this way with requests - Take a look at this selenium code.

To get rid of text and spaces you can do:

.get_text(strip=True).replace('Was: ','')

Example

from selenium import webdriver
from bs4 import BeautifulSoup
import time

url = "https://www.petshop.co.uk/Dog"
driver = webdriver.Chrome('C:Program FilesChromeDriverchromedriver.exe')
driver.get(url)
time.sleep(3)

html = driver.page_source
soup = BeautifulSoup(html,'html.parser')
for old_price in soup.find_all("small", class_ = "product-views-price-old"):
    print(old_price.get_text(strip=True).replace('Was: ',''))

driver.quit()

Output

￡2.20
￡18.61
￡27.00
￡38.39
￡38.39
￡20.65
￡1.30
￡67.99
￡20.65
￡1.30
￡54.95
￡30.99