python - BeautifulSoup - stripping out non-breaking spaces from HTML

Question

I am trying to scrape a number of 10K Risk Factor sections, e.g. https://www.sec.gov/Archives/edgar/data/1321502/000143774910004615/andain_10k-123106.htm

One of my problems is that I'm trying to match a few strings exactly (such as "Risk Factors") but sometimes there are several non-breaking spaces between the words

I am hoping I can just strip them out as they're not useful to me, so I tried:

url = 'https://www.sec.gov/Archives/edgar/data/1321502/000143774910004615/andain_10k-123106.htm'
page = requests.get(url)
soup = BeautifulSoup(page.text.replace("xa0"," "), 'html.parser')

And then searching the soup (in the usual way) to test whether it worked:

soup.find_all(string="ITEM 1A.xa0xa0RISK FACTORS")

But the output still contains non-breaking spaces, which it shouldn't:

Out[42]: ['ITEM 1A.xa0xa0RISK FACTORS']

What am I doing wrong?

question from:https://stackoverflow.com/questions/65829163/beautifulsoup-stripping-out-non-breaking-spaces-from-html

Answer 1

Try this:

import requests
from bs4 import BeautifulSoup

url = 'https://www.sec.gov/Archives/edgar/data/1321502/000143774910004615/andain_10k-123106.htm'
page = requests.get(url)
soup = BeautifulSoup(page.content, 'html.parser')
cleaned_up = [
    i.getText(strip=True).replace(u"xa0", " ")
    for i in soup.find_all("font") if i.getText().startswith("ITEM")
]
print(cleaned_up[1])

Output:

ITEM 1A.  RISK FACTORS