[Numpy-tickets] [NumPy] #479: MemoryError creating recarrays from medium sized python objects

NumPy numpy-tickets@scipy....
Mon Mar 26 12:25:16 CDT 2007

#479: MemoryError creating recarrays from medium sized python objects
 Reporter:  faltet  |       Owner:  somebody
     Type:  defect  |      Status:  new     
 Priority:  normal  |   Milestone:          
Component:  Other   |     Version:          
 Severity:  normal  |    Keywords:          
 The next exposes the (weird) behaviour:

 In [207]:N=8670
 In [208]:numpy.rec.array([[range(N)]], dtype=[('var1', 'f8', (N,))])
 recarray([ (array([  0.00000000e+00,   1.00000000e+00,   2.00000000e+00,
          8.66700000e+03,   8.66800000e+03,   8.66900000e+03]),)],
       dtype=[('var1', '<f8', (8670,))])
 In [209]:N=8680
 In [210]:numpy.rec.array([[range(N)]], dtype=[('var1', 'f8', (N,))])
 exceptions.MemoryError                               Traceback (most
 recent call last)

 /home/faltet/<ipython console>

 /usr/lib/python2.4/site-packages/numpy/core/records.py in array(obj,
 dtype, shape, offset, strides, formats, names, titles, aligned, byteorder,
     531     elif isinstance(obj, (list, tuple)):
     532         if isinstance(obj[0], (tuple, list)):
 --> 533             return fromrecords(obj, dtype=dtype, shape=shape,
     534         else:
     535             return fromarrays(obj, dtype=dtype, shape=shape,

 /usr/lib/python2.4/site-packages/numpy/core/records.py in
 fromrecords(recList, dtype, shape, formats, names, titles, aligned,
     385     try:
 --> 386         retval = sb.array(recList, dtype = descr)
     387     except TypeError:  # list of lists instead of list of tuples
     388         if (shape is None or shape == 0):


 The memory of my machine is 512 MB, and the recarray + python object
 should fit in far less than 1 MB.

 Curiously enough, you can achieve the desired recarray by using a regular
 numpy array

 In [211]:numpy.rec.array(numpy.arange(N*1.), dtype=[('var1', 'f8', (N,))])
 recarray([ (array([  0.00000000e+00,   1.00000000e+00,   2.00000000e+00,
          8.67700000e+03,   8.67800000e+03,   8.67900000e+03]),)],
       dtype=[('var1', '<f8', (8680,))])

 But note the strange nesting of the resulting recarray (see ticket #478).

Ticket URL: <http://projects.scipy.org/scipy/numpy/ticket/479>
NumPy <http://projects.scipy.org/scipy/numpy>
The fundamental package needed for scientific computing with Python.

More information about the Numpy-tickets mailing list