python - Using BeautifulSoup to return pandas dataframe

Question

I've recently started to play with BeautifulSoup in order to extract specific data in HTML and convert to a pandas dataframe. I've been stuck as the data I am returning is a string and not a list.

My goal is to get all info under the results tag in the HTML report in their original data type (list of dict) so I can iterate over and return a data frame.

Here is the code:

import requests
from bs4 import BeautifulSoup

countriy_code = 'BE'
city = 'BRUSSELS'.upper()
code_postal = '1000'
max_price = '250000'

URL = f'https://www.immoweb.be/en/search/apartment/for-sale?countries={countriy_code}&districts={city}&postalCodes={code_postal}&maxPrice={max_price}&orderBy=relevance'
print(URL)

page = requests.get(URL)
soup = BeautifulSoup(page.content, 'html.parser')
page_results = soup.find('iw-search')
page_attributes = page_results.attrs

for key, items in page_attributes.items():
    print(type(key), type(items))

The raw HTML looks like (note it's not the full message):

<iw-search :anchor-card-id="null" :criteria='{"countries":"BE","districts":[{"queryValue":"BRUSSELS","queryParam":"districts","slug":"brussels","shortLabel":"Brussels (District)","label":"Brussels (District)","translations":{"fr":"Bruxelles","en":"Brussels","nl":"Brussel"}}],"maxPrice":250000,"postalCodes":[{"queryValue":"1000","queryParam":"postalCodes","slug":"brussels-city","shortLabel":"Brussels City (1000)","label":"Brussels City (1000)","translations":{"fr":"Bruxelles ville","en":"Brussels City","nl":"Brussel"}}],"propertyTypes":"APARTMENT","transactionTypes":"FOR_SALE"}' :geo-point-count="876" :labellized-search='"Apartment for sale - Brussels (District) (and 1 more)"' :marketing-count="1457" :page="1" :result-count="1457" :results='[{"id":9103642,"cluster":{"minPrice":null,"maxPrice":null,"minRoom":null,"maxRoom":null,"minSurface":null,"maxSurface":null,"projectInfo":null,"bedroomRange":"","surfaceRange":""},"customerLogoUrl":"https://static.immoweb.be/logos/145409.gif?cache=2016051503060","customerName":"Expertissimmo","flags":{"main":"under_option","secondary":[],"percentSold":null},"media":{"pictures":[{"smallUrl":"https://static.immoweb.be/photos/0/9/1/0/3/6/4/2/9103642_1.gif?cache=20210106041435","mediumUrl":"https://static.immoweb.be/photos/0/9/1/0/3/6/4/2/M_9103642_1.jpg?cache=20210106041435","largeUrl":"https://static.immoweb.be/photos/0/9/1/0/3/6/4/2/9103642_1.jpg?cache=20210106041435","isVertical":false},{"smallUrl":"https://static.immoweb.be/photos/0/9/1/0/3/6/4/2/9103642_2.gif?cache=20210106041435","mediumUrl":```

thanks for your return

question from:https://stackoverflow.com/questions/65649670/using-beautifulsoup-to-return-pandas-dataframe

Answer 1

You can get a list of dictionary objects using json (the string is valid json)

import json

res = json.loads(page_attributes[':results'])