[Scipy-tickets] [SciPy] #1371: MemoryError in scipy.io.savemat

SciPy Trac scipy-tickets@scipy....
Mon Jan 24 08:08:44 CST 2011


#1371: MemoryError in scipy.io.savemat
--------------------------------------------------+-------------------------
 Reporter:  Cerno                                 |       Owner:  somebody
     Type:  defect                                |      Status:  new     
 Priority:  normal                                |   Milestone:  0.9.0   
Component:  scipy.io                              |     Version:  0.8.0   
 Keywords:  matlab io savemat memory MemoryError  |  
--------------------------------------------------+-------------------------
 The following code produces a MemoryError when trying to save a numpy
 matrix to Matlab format:

 import numpy as np

 import scipy

 import scipy.io

 vec1 = np.array(range(20000))

 vec2 = vec1.repeat(10000)

 scipy.io.savemat('c:\\test1.mat', mdict={'d' : vec1})

 scipy.io.savemat('c:\\test2.mat', mdict={'d' : vec2})

 Note that you might have to vary the range parameters to reproduce this.
 They have to be large enough to produce the memory error but not so large
 that vec2 does not fit into memory.

 The complete error message is this:

 Traceback (most recent call last):
   File "pytest.py", line 9, in <module>
     scipy.io.savemat('c:\\test2.mat', mdict={'d' : vec2})
   File "C:\Tools\common\python\python254\lib\site-
 packages\scipy\io\matlab\mio.py", line 199, in savemat
     MW.put_variables(mdict)
   File "C:\Tools\common\python\python254\lib\site-
 packages\scipy\io\matlab\mio5.py", line 900, in put_variables
     self._matrix_writer.write_top(var, name, is_global)
   File "C:\Tools\common\python\python254\lib\site-
 packages\scipy\io\matlab\mio5.py", line 654, in write_top
     self.write(arr)
   File "C:\Tools\common\python\python254\lib\site-
 packages\scipy\io\matlab\mio5.py", line 691, in write
     self.write_numeric(narr)
   File "C:\Tools\common\python\python254\lib\site-
 packages\scipy\io\matlab\mio5.py", line 711, in write_numeric
     self.write_element(arr)
   File "C:\Tools\common\python\python254\lib\site-
 packages\scipy\io\matlab\mio5.py", line 564, in write_element
     self.write_regular_element(arr, mdtype, byte_count)
   File "C:\Tools\common\python\python254\lib\site-
 packages\scipy\io\matlab\mio5.py", line 580, in write_regular_element
     self.write_bytes(arr)
   File "C:\Tools\common\python\python254\lib\site-
 packages\scipy\io\matlab\mio5.py", line 551, in write_bytes
     self.file_stream.write(arr.tostring(order='F'))
 MemoryError

 From my limited understanding of the code, scipy constructs the matlab
 file completely in memory and then dumps it to the harddrive. This
 effectively doubles the amount of memory used by the vector and seems to
 cause the crash. If possible, it might be more efficient to construct the
 mat file byte by byte and stream it to disk, so that only little
 additional memory is used.

-- 
Ticket URL: <http://projects.scipy.org/scipy/ticket/1371>
SciPy <http://www.scipy.org>
SciPy is open-source software for mathematics, science, and engineering.


More information about the Scipy-tickets mailing list