python - How to create a numpy array of lists?

Question

I want to create a numpy array in which each element must be a list, so later I can append new elements to each.

I have looked on google and here on stack overflow already, yet it seems nowhere to be found.

Main issue is that numpy assumes your list must become an array, but that is not what I am looking for.

Answer 1

As you discovered, np.array tries to create a 2d array when given something like

 A = np.array([[1,2],[3,4]],dtype=object)

You have apply some tricks to get around this default behavior.

One is to make the sublists variable in length. It can't make a 2d array from these, so it resorts to the object array:

In [43]: A=np.array([[1,2],[],[1,2,3,4]])
In [44]: A
Out[44]: array([[1, 2], [], [1, 2, 3, 4]], dtype=object)

And you can then append values to each of those lists:

In [45]: for i in A: i.append(34)
In [46]: A
Out[46]: array([[1, 2, 34], [34], [1, 2, 3, 4, 34]], dtype=object)

np.empty also creates an object array:

In [47]: A=np.empty((3,),dtype=object)
In [48]: A
Out[48]: array([None, None, None], dtype=object)

But you then have to be careful how you change the elements to lists. np.fill is tempting, but has problems:

In [49]: A.fill([])
In [50]: A
Out[50]: array([[], [], []], dtype=object)
In [51]: for i in A: i.append(34)
In [52]: A
Out[52]: array([[34, 34, 34], [34, 34, 34], [34, 34, 34]], dtype=object)

It turns out that fill puts the same list in all slots, so modifying one modifies all the others. You can get the same problem with a list of lists:

In [53]: B=[[]]*3
In [54]: B
Out[54]: [[], [], []]
In [55]: for i in B: i.append(34)
In [56]: B
Out[56]: [[34, 34, 34], [34, 34, 34], [34, 34, 34]]

The proper way to initial the empty A is with an iteration, e.g.

In [65]: A=np.empty((3,),dtype=object)
In [66]: for i,v in enumerate(A): A[i]=[v,i]
In [67]: A
Out[67]: array([[None, 0], [None, 1], [None, 2]], dtype=object)
In [68]: for v in A: v.append(34)
In [69]: A
Out[69]: array([[None, 0, 34], [None, 1, 34], [None, 2, 34]], dtype=object)

It's a little unclear from the question and comments whether you want to append to the lists, or append lists to the array. I've just demonstrated appending to the lists.

There is an np.append function, which new users often misuse. It isn't a substitute for list append. It is a front end to np.concatenate. It is not an in-place operation; it returns a new array.

Also defining a list to add with it can be tricky:

In [72]: np.append(A,[[1,23]])
Out[72]: array([[None, 0, 34], [None, 1, 34], [None, 2, 34], 1, 23],     dtype=object)

You need to construct another object array to concatenate to the original, e.g.

In [76]: np.append(A,np.empty((1,),dtype=object))
Out[76]: array([[None, 0, 34], [None, 1, 34], [None, 2, 34], None], dtype=object)

In all of this, an array of lists is harder to construct than a list of lists, and no easier, or faster, to manipulate. You have to make it a 2d array of lists to derive some benefit.

In [78]: A[:,None]
Out[78]: 
array([[[None, 0, 34]],
       [[None, 1, 34]],
       [[None, 2, 34]]], dtype=object)

You can reshape, transpose, etc an object array, where as creating and manipulating a list of lists of lists gets more complicated.

In [79]: A[:,None].tolist()
Out[79]: [[[None, 0, 34]], [[None, 1, 34]], [[None, 2, 34]]]

===

As shown in https://stackoverflow.com/a/57364472/901925, np.frompyfunc is a good tool for creating an array of objects.

np.frompyfunc(list, 0, 1)(np.empty((3,2), dtype=object))