python 2.7 - Pandas Dataframe ValueError: Shape of passed values is (X, ), indices imply (X, Y)

Question

I am getting an error and I'm not sure how to fix it.

The following seems to work:

def random(row):
   return [1,2,3,4]

df = pandas.DataFrame(np.random.randn(5, 4), columns=list('ABCD'))

df.apply(func = random, axis = 1)

and my output is:

[1,2,3,4]
[1,2,3,4]
[1,2,3,4]
[1,2,3,4]

However, when I change one of the of the columns to a value such as 1 or None:

def random(row):
   return [1,2,3,4]

df = pandas.DataFrame(np.random.randn(5, 4), columns=list('ABCD'))
df['E'] = 1

df.apply(func = random, axis = 1)

I get the the error:

ValueError: Shape of passed values is (5,), indices imply (5, 5)

I've been wrestling with this for a few days now and nothing seems to work. What is interesting is that when I change

def random(row):
   return [1,2,3,4]

to

def random(row):
   print [1,2,3,4]

everything seems to work normally.

This question is a clearer way of asking this question, which I feel may have been confusing.

My goal is to compute a list for each row and then create a column out of that.

EDIT: I originally start with a dataframe that hase one column. I add 4 columns in 4 difference apply steps, and then when I try to add another column I get this error.

Answer 1

If your goal is add new column to DataFrame, just write your function as function returning scalar value (not list), something like this:

>>> def random(row):
...     return row.mean()

and then use apply:

>>> df['new'] = df.apply(func = random, axis = 1)
>>> df
          A         B         C         D       new
0  0.201143 -2.345828 -2.186106 -0.784721 -1.278878
1 -0.198460  0.544879  0.554407 -0.161357  0.184867
2  0.269807  1.132344  0.120303 -0.116843  0.351403
3 -1.131396  1.278477  1.567599  0.483912  0.549648
4  0.288147  0.382764 -0.840972  0.838950  0.167222

I don't know if it possible for your new column to contain lists, but it deinitely possible to contain tuples ((...) instead of [...]):

>>> def random(row):
...    return (1,2,3,4,5)
...
>>> df['new'] = df.apply(func = random, axis = 1)
>>> df
          A         B         C         D              new
0  0.201143 -2.345828 -2.186106 -0.784721  (1, 2, 3, 4, 5)
1 -0.198460  0.544879  0.554407 -0.161357  (1, 2, 3, 4, 5)
2  0.269807  1.132344  0.120303 -0.116843  (1, 2, 3, 4, 5)
3 -1.131396  1.278477  1.567599  0.483912  (1, 2, 3, 4, 5)
4  0.288147  0.382764 -0.840972  0.838950  (1, 2, 3, 4, 5)