Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
918 views
in Technique[技术] by (71.8m points)

python - AssertionError: 22 columns passed, passed data had 21 columns

This is my code:

from urllib import urlopen
from bs4 import BeautifulSoup
import pandas as pd

url = "http://www.basketball-reference.com/draft/NBA_2014.html"
html = urlopen(url)
soup = BeautifulSoup(html)
column_headers = [th.getText() for th in soup.findAll('tr',limit=2)[1].findAll('th')]
data_rows = soup.findAll('tr')[2:]
player_data = [[td.getText() for td in data_rows[i].findAll('td')] for i in range(len(data_rows))] #PLAYER DATA 

type(soup)
type(data_rows)

df = pd.DataFrame(player_data,columns=column_headers)

The error seems to occur in the last line.

See Question&Answers more detail:os

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Reply

0 votes
by (71.8m points)

First of all, the error is pretty straight-forward: your column_headers list has 22 columns, but player_data entries only have 21. So you need to find which out column is missing and why. Just by visually comparing the entries from the dataframe and the headers list, it appears one of the two first columns is missing. player_data[0][0] returns

1, CLE, Andrew Wiggins, University of Kansas,... but it should be

1, 1, CLE, Andrew Wiggins, University of Kansas,...

The problem is the table itself. Navigate to the website, hover over the table and right-click: inspect.

The first row of data (underneath the 'Rk') consists of 21 td and 1 th element. The "rk" entry is actually of type th and not td:

Screenshot of table of provided data

That is why your

player_data = [[td.getText() for td in data_rows[i].findAll('td')] for i in range(len(data_rows))] 

skips the first column because it is only iterating over td elements. Hence the different length. I don't know how important the first column is; quick fix would be to drop the Rk column from your headers list.

Alternatively, search for both td and th elements:

player_data = [[td.getText() for td in data_rows[i].findAll(['td','th'])] for i in range(len(data_rows))]

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
OGeek|极客中国-欢迎来到极客的世界,一个免费开放的程序员编程交流平台!开放,进步,分享!让技术改变生活,让极客改变未来! Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...