python - I can't get the a tag using Beautifulsoup, though I can get other tags

Question

I'm just trying to get data from a webpage called "Elgiganten" and its url: https://www.elgiganten.se/

I want to get the products name and its url. When I tried to get the a tag so I got an empty list, but I could get the span tag though taht they were in the same div tag.

Here is the whole code:

from bs4 import BeautifulSoup
import requests


respons = requests.get("https://www.elgiganten.se")

soup = BeautifulSoup(respons.content, "lxml")

g_data = soup.find_all("div", {"class": "col-flex S-order-1"})

for item in g_data:
    print(item.contents[1].find_all("span")[0])
    print(item.contents[1].find_all("a", {"class": "product-name"}))

I hope that anyone can tell me why the a tag seems to be invisible, and can fix the issue.

question from:https://stackoverflow.com/questions/65926630/i-cant-get-the-a-tag-using-beautifulsoup-though-i-can-get-other-tags

Answer 1

Go for the a-tags directly. You can extract the product name and the url both from that tag:

from bs4 import BeautifulSoup
import requests

respons = requests.get("https://www.elgiganten.se")
soup = BeautifulSoup(respons.content, "lxml")

g_data = soup.find_all("a", {"class": "product-name"}, href=True)

for item in g_data:
  print(item['title'], item['href'])