python - How do I retrieve objects based on a partial custom attribute name in Scrapy?

Question

I have the following element:

<div data-offer="MTs3O29sZG5hdnkuY29tOzQxMDYy" class="Offer__Card-sc-14rx0hy-0 iBdrTi"></div>

I need to find it with scrapy but I have two comlpications. The class can change so it is not going to have that value. Pretty much off the table.

The second problem is that data-offer value can vary between data-offer, data-offer-promo, data-offer-double

Do you know how can I find this elements based on a partial attribute name? Like bring me everything that has a custom attribute "data-offer*" Or everything that starts with it works too, but not the value, the attribute name.

I tried this with no success

 response.css('[div::attr^="data-offer"]')

question from:https://stackoverflow.com/questions/65919729/how-do-i-retrieve-objects-based-on-a-partial-custom-attribute-name-in-scrapy

Answer 1

You can find those elements using beautifulSoup. This will find the first div element that has a "data-offer" attribute:

soup = BeautifulSoup(response.body, 'lxml')
results = soup.find("div", {"data-offer" : True})

You could also get a list with all the elements that have the same condition:

soup = BeautifulSoup(response.body, 'lxml')
results = soup.find_all("div", {"data-offer" : True})