pandas - How do I get my parsed data to fall under certain columns? [python]

Question

Here is my code. I've scraped the data from website but it just returns to me one long list.

How do I manipulate the data to fall under the headings? I'm getting the current error message:

ValueError: 8 columns passed, passed data had 2648 columns.

Any help is greatly appreciated.

from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
from bs4 import BeautifulSoup
import pandas as pd
from pandas import DataFrame
import html5lib

url = "https://www.loudnumber.com/screeners/cashflow"
driver = webdriver.Chrome(ChromeDriverManager().install())
driver.get(url)


time.sleep(5)

html = driver.page_source
soup = BeautifulSoup(html,'html.parser')
table = soup.find('table')
l = []
for tr in table:
    td = table.find_all('td') #cols
    rows = [table.text.strip() for tr in td if tr.text.strip()] 
    if rows:
        l.append(rows)
                            

driver.quit()


df = pd.DataFrame(list(l), columns=["Ticker","Company","Industry","Current Price"
                                    ,"Instrinsic Value","IV to CP ratio",
                                    "Dividend","Dividend Yield"])

print(df)

question from:https://stackoverflow.com/questions/65872117/how-do-i-get-my-parsed-data-to-fall-under-certain-columns-python

Answer 1

Lots of things are totally wrong here, so fixed your code.

Note: Put selenium drivers in python folder to make this code work or give its path. refer this selenium - chromedriver executable needs to be in PATH

Here is complete code:

from selenium import webdriver
from bs4 import BeautifulSoup
import pandas as pd
import time

url = "https://www.loudnumber.com/screeners/cashflow"
driver = webdriver.Chrome()
driver.get(url)
time.sleep(3)

html = driver.page_source
soup = BeautifulSoup(html,'html.parser')
table = soup.find("div", class_="table-responsive").tbody.find_all("tr")
l = []

for tr in table:
    td = tr.find_all('td') #cols
    rows = [i.text.strip() for i in td if i.text.strip()]
    l.append(rows)

driver.quit()

df = pd.DataFrame(list(l), columns=["Ticker","Company","Industry","Current Price" ,"Instrinsic Value","IV to CP ratio","Dividend","Dividend Yield"])
print(df)

Here is the result: