[Numpy-tickets] [NumPy] #479: MemoryError creating recarrays from medium sized python objects

[NumPy]

#479: MemoryError creating recarrays from medium sized python objects
--------------------+-------------------------------------------------------
 Reporter:  faltet  |       Owner:  somebody
     Type:  defect  |      Status:  new     
 Priority:  normal  |   Milestone:          
Component:  Other   |     Version:          
 Severity:  normal  |    Keywords:          
--------------------+-------------------------------------------------------
 The next exposes the (weird) behaviour:

 {{{
 In [207]:N=8670
 In [208]:numpy.rec.array([[range(N)]], dtype=[('var1', 'f8', (N,))])
 Out[208]:
 recarray([ (array([  0.00000000e+00,   1.00000000e+00,   2.00000000e+00,
 ...,
          8.66700000e+03,   8.66800000e+03,   8.66900000e+03]),)],
       dtype=[('var1', '<f8', (8670,))])
 In [209]:N=8680
 In [210]:numpy.rec.array([[range(N)]], dtype=[('var1', 'f8', (N,))])
 ---------------------------------------------------------------------------
 exceptions.MemoryError                               Traceback (most
 recent call last)

 /home/faltet/<ipython console>

 /usr/lib/python2.4/site-packages/numpy/core/records.py in array(obj,
 dtype, shape, offset, strides, formats, names, titles, aligned, byteorder,
 copy)
     531     elif isinstance(obj, (list, tuple)):
     532         if isinstance(obj[0], (tuple, list)):
 --> 533             return fromrecords(obj, dtype=dtype, shape=shape,
 **kwds)
     534         else:
     535             return fromarrays(obj, dtype=dtype, shape=shape,
 **kwds)

 /usr/lib/python2.4/site-packages/numpy/core/records.py in
 fromrecords(recList, dtype, shape, formats, names, titles, aligned,
 byteorder)
     384
     385     try:
 --> 386         retval = sb.array(recList, dtype = descr)
     387     except TypeError:  # list of lists instead of list of tuples
     388         if (shape is None or shape == 0):

 MemoryError:
 }}}

 The memory of my machine is 512 MB, and the recarray + python object
 should fit in far less than 1 MB.

 Curiously enough, you can achieve the desired recarray by using a regular
 numpy array

 {{{
 In [211]:numpy.rec.array(numpy.arange(N*1.), dtype=[('var1', 'f8', (N,))])
 Out[211]:
 recarray([ (array([  0.00000000e+00,   1.00000000e+00,   2.00000000e+00,
 ...,
          8.67700000e+03,   8.67800000e+03,   8.67900000e+03]),)],
       dtype=[('var1', '<f8', (8680,))])
 }}}

 But note the strange nesting of the resulting recarray (see ticket #478).

-- 
Ticket URL: <http://projects.scipy.org/scipy/numpy/ticket/479>
NumPy <http://projects.scipy.org/scipy/numpy>
The fundamental package needed for scientific computing with Python.