python - Adding datetime field to recarray

Question

I am trying to append a date-time field (datetime64) to an existing recarray - without much success. I can create the datetime field, but when I attempt to append it to the record array I get the error:

ValueError: Error parsing datetime string "?" at position 0

However, if I cast the data as int64 I can add it in that format without problem. (code shown below)

Anyone know why this does not work?

(my ultimate goal is to write the recarray to a netcdf file, so an appropriate datetime format with that goal in mind would also be helpful)

I am using python 2.7.6.1, numpy 1.8.1

Thanks, Rob

import numpy as np
import numpy.lib.recfunctions as rf

# ----- make a recarray ---------    
dummy = np.arange(0,10)
datarray = np.core.records.fromarrays([dummy,dummy,dummy],names='a,b,c')

# ----- make some time data using datetime64 ---------
sec = np.arange(0,10)*1000
millisec = np.arange(0,10) 
mytime = sec + millisec

mytime64 = mytime.astype('timedelta64[ms]')         
basetime =  np.datetime64('1990-01-01') 
mydatetime = mytime64+basetime  

# ----- convert time data to int64 ---------
idatetime = mydatetime.astype('int64');

#------ try and append to recarray ---------
#  this works
datarray = rf.append_fields(datarray, 'iDateTime', data=idatetime)
# this doesnt
datarray = rf.append_fields(datarray, 'DateTime', data=mydatetime)

Answer 1

The traceback is:

Traceback (most recent call last):
  File "stack26739733.py", line 30, in <module>
    datarray = rf.append_fields(datarray, 'DateTime', data=mydatetime, usemask=False, dtypes=mydatetime.dtype)
  File "/usr/local/lib/python2.7/site-packages/numpy/lib/recfunctions.py", line 641, in append_fields
    dtype=base.dtype.descr + data.dtype.descr)
  File "/usr/local/lib/python2.7/site-packages/numpy/ma/extras.py", line 163, in masked_all
    mask=np.ones(shape, make_mask_descr(dtype)))
  File "/usr/local/lib/python2.7/site-packages/numpy/ma/core.py", line 2644, in __new__
    _data = ndarray.view(_data, cls)
  File "/usr/local/lib/python2.7/site-packages/numpy/ma/core.py", line 2800, in __array_finalize__
    self._fill_value = _check_fill_value(None, self.dtype)
  File "/usr/local/lib/python2.7/site-packages/numpy/ma/core.py", line 402, in _check_fill_value
    dtype=ndtype,)
ValueError: Error parsing datetime string "?" at position 0

So this append function constructs a masked array (ma), and checks the 'fill_value' for the appended 'dtype'. Apparently _check_fill_value doesn't understand the datetime dtype. Looks like an incompatibility between masked array and datetime. A numpy bug report might be in order.

---

Here's a simple, do-it-yourself append:

dt1 = np.dtype(datarray.dtype.descr + mydatetime.dtype.descr)
newarray = np.empty(datarray.shape, dtype=dt1)
for n in datarray.dtype.names:
    newarray[n] = datarray[n]
newarray['f3'] = mydatetime

I construct an empty array with a union dtype. Then I copy the data from both datarray and mydatetime field by field. Since the number of fields is normally quite small compared to the shape, this copy is quite fast. I'm pretty sure the rf function does the same.

'f3' is the default name of the added field. You can change that when creating dt1.

The result is:

array([(0, 0, 0, datetime.datetime(1990, 1, 1, 0, 0)),
       (1, 1, 1, datetime.datetime(1990, 1, 1, 0, 0, 1, 1000)),
       (2, 2, 2, datetime.datetime(1990, 1, 1, 0, 0, 2, 2000)),
       ...
       (9, 9, 9, datetime.datetime(1990, 1, 1, 0, 0, 9, 9000))], 
      dtype=[('a', '<i4'), ('b', '<i4'), ('c', '<i4'), ('f3', '<M8[ms]')])

Turning this newarray into a masked array produces the same _check_fill_value error.

np.ma.masked_array(newarray)