[Numpy-tickets] [NumPy] #897: Numpy mrecords - unpickle causes unhandled win32 exception

[NumPy]

#897: Numpy mrecords - unpickle causes unhandled win32 exception
-----------------------------+----------------------------------------------
 Reporter:  chrisshucksmith  |       Owner:  pierregm
     Type:  defect           |      Status:  new     
 Priority:  normal           |   Milestone:  1.2.0   
Component:  numpy.ma         |     Version:  none    
 Severity:  major            |    Keywords:          
-----------------------------+----------------------------------------------
 Putting a datetime object into a masked record array (causing one or more
 entries in the dtype to be '|O4') seems to trigger a bug in unpickling,
 resulting in either a TypeError "object pickle not returning list" or
 sudden-death python.exe crash.

 '''Crash:'''
 {{{
 import os
 from pickle import Pickler, Unpickler, HIGHEST_PROTOCOL
 from StringIO import StringIO
 from numpy import array, ones, zeros
 from numpy.ma import mrecords
 from datetime import datetime

 cache_dir = os.curdir

 rows = []
 rowmasks = []
 for i in xrange(5):
     x = datetime(2008,6,5)
     rows.append([x,x,x,x,x])
     mask = zeros([5], 'bool')
     mask[2] = True
     rowmasks.append(mask)

 print 'data:\n', rows
 print 'Mask:\n', rowmasks

 recarr = mrecords.fromrecords(rows, names=['Ones','Twos', 'Threes',
 'Fours', 'Fives'], mask=rowmasks)

 print 'Records;'
 print recarr.dtype
 print recarr[0]
 print recarr.Twos
 print recarr.Threes

 print 'Picking'
 sio = StringIO()
 p = Pickler(sio, HIGHEST_PROTOCOL)
 p.dump(recarr)
 print 'Unpickling'
 sio.seek(0)
 u = Unpickler(sio)
 recarr2 = u.load()
 print recarr2[0]
 }}}


 '''Exception case:'''

 {{{
 Traceback (most recent call last):
   File "C:\workspace\eclipse\hacking\src\pycrash.py", line 33, in <module>
     recarr2 = u.load()
   File "c:\python25\lib\pickle.py", line 858, in load
     dispatch[key](self)
   File "c:\python25\lib\pickle.py", line 1217, in load_build
     setstate(state)
   File "C:\Python25\Lib\site-packages\numpy\ma\mrecords.py", line 520, in
 __setstate__
     ndarray.__setstate__(self, (shp, typ, isf, raw))
 TypeError: object pickle not returning list
 }}}

 Code
 {{{
 import os
 from pickle import Pickler, Unpickler, HIGHEST_PROTOCOL
 from StringIO import StringIO
 from numpy import array, ones, zeros
 from numpy.ma import mrecords
 from datetime import datetime

 cache_dir = os.curdir

 rows = []
 rowmasks = []
 for i in xrange(6):
     rows.append( [ datetime(2008,6,5) ] )
     rowmasks.append( zeros([1], 'bool'))

 print 'data:\n', rows
 print 'Mask:\n', rowmasks

 recarr = mrecords.fromrecords(rows, names=['Dates'], mask=rowmasks)

 print 'Records;'
 print recarr.dtype
 print recarr[0]
 print recarr.Dates

 print 'Picking'
 sio = StringIO()
 p = Pickler(sio, HIGHEST_PROTOCOL)
 p.dump(recarr)
 print 'Unpickling'
 sio.seek(0)
 u = Unpickler(sio)
 recarr2 = u.load()
 print recarr2[0]
 print recarr2[1]
 }}}

 I'm aware neither of these cases represent good code, the datetime should
 concisely be stored in a numerical representation. Tested with python 2.5
 and various Numpy versions including numpy-1.1.1-py2.5.egg

-- 
Ticket URL: <http://scipy.org/scipy/numpy/ticket/897>
NumPy <http://projects.scipy.org/scipy/numpy>
The fundamental package needed for scientific computing with Python.