Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
422 views
in Technique[技术] by (71.8m points)

python - Retrieve indices of NaN values in a pandas dataframe

I try to retrieve for each row containing NaN values all the indices of the corresponding columns.

d=[[11.4,1.3,2.0, NaN],[11.4,1.3,NaN, NaN],[11.4,1.3,2.8, 0.7],[NaN,NaN,2.8, 0.7]]
df = pd.DataFrame(data=d, columns=['A','B','C','D'])
print df

      A    B    C    D
0  11.4  1.3  2.0  NaN
1  11.4  1.3  NaN  NaN
2  11.4  1.3  2.8  0.7
3  NaN   NaN  2.8  0.7

I've already done the following :

  • add a column with the count of NaN for each row
  • get the indices of each row containing NaN values

What I want (ideally the name of the column) is get a list like this :

[ ['D'],['C','D'],['A','B'] ]

Hope I can find a way without doing for each row the test for each column

if df.ix[i][column] == NaN:

I'm looking for a pandas way to be able to deal with my huge dataset.

Thanks in advance.

See Question&Answers more detail:os

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Reply

0 votes
by (71.8m points)

It should be efficient to use a scipy coordinate-format sparse matrix to retrieve the coordinates of the null values:

import scipy.sparse as sp

x,y = sp.coo_matrix(df.isnull()).nonzero()
print(list(zip(x,y)))

[(0, 3), (1, 2), (1, 3), (3, 0), (3, 1)]

Note that I'm calling the nonzero method in order to just output the coordinates of the nonzero entries in the underlying sparse matrix since I don't care about the actual values which are all True.


与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
OGeek|极客中国-欢迎来到极客的世界,一个免费开放的程序员编程交流平台!开放,进步,分享!让技术改变生活,让极客改变未来! Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...