python - Could not scrape some image links from a webpage using requests module

Question

I've created a script using requests and BeautifulSoup library to parse the links of some images from a webpage. The image links are visible when you use this selector [class^='cylindo-viewer-frame'] > img[src*='/frames/'] within the search bar (Ctrl + F) after inspecting element. This how they look like in the dom.

I know I can grab those image links using selenium but I would like to stick with requests module. I've noticed several times that there are always possibilities to grab such dynamic content using requests module. I've tried finding those links within script tag and in dev tools but no luck.

Two of the expected links out of 32 are:

https://content.cylindo.com/api/v2/4616/products/657285/frames/5/657285.JPG?background=FFFFFF&feature=FABRIC:Q1031&size=1268
https://content.cylindo.com/api/v2/4616/products/657285/frames/7/657285.JPG?background=FFFFFF&feature=FABRIC:Q1031&size=1268

This is how I've tried:

import requests
from bs4 import BeautifulSoup

link = 'https://www.ethanallen.com/on/demandware.store/Sites-ethanallen-us-Site/en_US/Product-Variation?pid=emersonQS&dwvar_emersonQS_Fabric=Q1031&dwvar_emersonQS_seatingSize=90sofa&step=2'

with requests.Session() as s:
    s.headers['User-Agent'] = 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36'
    r = s.get(link)
    soup = BeautifulSoup(r.text,"lxml")
    for item in soup.select(".cylindo-viewer-container li[class^='cylindo-viewer-frame'] > img[src*='/frames/']"):
        print(item.get("src"))

How can I grab those image links using requests?

question from:https://stackoverflow.com/questions/65830252/could-not-scrape-some-image-links-from-a-webpage-using-requests-module

Answer 1

Why you should use selenium?

Website serves content dynamically, what is not to handle with requests, cause the information you try to match is not in the response.

Take a look, it is not that hard ;)

Example

from selenium import webdriver
from bs4 import BeautifulSoup
from time import sleep

driver = webdriver.Chrome(executable_path='C:Program FilesChromeDriverchromedriver.exe')
url = "https://www.ethanallen.com/on/demandware.store/Sites-ethanallen-us-Site/en_US/Product-Variation?pid=emersonQS&dwvar_emersonQS_Fabric=Q1031&dwvar_emersonQS_seatingSize=90sofa&step=2"

driver.get(url)
sleep(2)

soup = BeautifulSoup(driver.page_source, 'lxml')

for item in soup.select(".cylindo-viewer-container li[class^='cylindo-viewer-frame'] > img[src*='/frames/']"):
        print(item.get("src"))
    
driver.close()

Output

https://content.cylindo.com/api/v2/4616/products/657285/frames/3/657285.JPG?background=FFFFFF&feature=FABRIC:Q1031&size=1268
https://content.cylindo.com/api/v2/4616/products/657285/frames/27/657285.JPG?background=FFFFFF&feature=FABRIC:Q1031&size=1268
https://content.cylindo.com/api/v2/4616/products/657285/frames/29/657285.JPG?background=FFFFFF&feature=FABRIC:Q1031&size=1268
https://content.cylindo.com/api/v2/4616/products/657285/frames/11/657285.JPG?background=FFFFFF&feature=FABRIC:Q1031&size=1268
https://content.cylindo.com/api/v2/4616/products/657285/frames/31/657285.JPG?background=FFFFFF&feature=FABRIC:Q1031&size=1268
https://content.cylindo.com/api/v2/4616/products/657285/frames/5/657285.JPG?background=FFFFFF&feature=FABRIC:Q1031&size=1268
...