[NumPy-Tickets] [NumPy] #1415: data corruption unpickling a numpy.void

[NumPy Trac]

#1415: data corruption unpickling a numpy.void
------------------------+---------------------------------------------------
 Reporter:  mspacek     |       Owner:  somebody
     Type:  defect      |      Status:  new     
 Priority:  high        |   Milestone:          
Component:  numpy.core  |     Version:  1.4.0   
 Keywords:              |  
------------------------+---------------------------------------------------
 See thread at
 http://article.gmane.org/gmane.comp.python.numeric.general/36611

 Pickling and then unpickling an entry of <type 'numpy.void'> or <type
 'numpy.record'> from a structured array or recarray results in data
 corruption on the round trip:

 {{{
 >>> import numpy as np
 >>> x = np.zeros((2,), dtype=('i4,f4,a10'))
 >>> x[:] = [(1,2.,'Hello'), (2,3.,"World")]
 >>> x
 array([(1, 2.0, 'Hello'), (2, 3.0, 'World')],
        dtype=[('f0', '<i4'), ('f1', '<f4'), ('f2', '|S10')])
 >>> x[0]
 (1, 2.0, 'Hello')
 >>> type(x[0])
 <type 'numpy.void'>
 >>> import pickle
 >>> s = pickle.dumps(x[0])
 >>> newx0 = pickle.loads(s)
 >>> newx0
 (30917960, 1.6904535998413144e-38, '\xd0\xef\x1c\x1eZ\x03\x00d')
 >>> s
 "cnumpy.core.multiarray\nscalar\np0\n(cnumpy\ndtype\np1\n(S'V18'
 \np2\nI0\nI1\ntp3\nRp4\n(I4\nS'|'\np5\nN(S'f0'\np6\nS'f1'\np7\nS'f2'
 \np8\ntp9\n(dp10\ng6\n(g1\n(S'i4'\np11\nI0\nI1\ntp12\nRp13
 \n(I4\nS'<'\np14\nNNNI-1\nI-1\nI0\nNtp15\nbI0\ntp16\nsg7\n(g1
 \n(S'f4'\np17\nI0\nI1\ntp18\nRp19\n(I4\nS'<'\np20\nNNNI-1\nI-1
 \nI0\nNtp21\nbI4\ntp22\nsg8\n(g1\n(S'S10'\np23\nI0\nI1\ntp24\nRp25
 \n(I4\nS'|'\np26\nNNNI10\nI1\nI0\nNtp27\nbI8\ntp28\nsI18\nI1\nI0
 \nNtp29\nbS'\\x01\\x00\\x00\\x00\\x00\\x00\\x00@Hello\\x00\\x00
 \\x00\\x00\\x00'\np30\ntp31\nRp32\n."
 >>> type(newx0)
 <type 'numpy.void'>
 >>> newx0.dtype
 dtype([('f0', '<i4'), ('f1', '<f4'), ('f2', '|S10')])
 >>> x[0].dtype
 dtype([('f0', '<i4'), ('f1', '<f4'), ('f2', '|S10')])
 >>> np.version.version
 '1.4.0'
 }}}

 It seems that the pickling works fine, but there's corruption on
 unpickling.

 Robert Kern says: "The implementation of numpy.core.multiarray.scalar is
 doing something wrong."

 Pauli Virtanen says: "The relevant code path to look at is
 multiarraymodule:array_scalar -> scalarapi.c:PyArray_Scalar. Needs some
 cgdb'ing to find out what's going on there."

 A nice temporary workaround is to convert the original structured array
 "x" into a list of scalar arrays, each of whose entries then
 pickle/unpickle correctly, and still allow dict-style named field access:

 {{{
 >>> x2 = map(np.asarray, x)
 }}}

-- 
Ticket URL: <http://projects.scipy.org/numpy/ticket/1415>
NumPy <http://projects.scipy.org/numpy>
My example project