[SciPy-User] memory errors when using savemat
Sun Jul 25 12:13:57 CDT 2010
On Jul 25, 2010, at 9:49 AM, Pauli Virtanen wrote:
> Sat, 24 Jul 2010 10:37:33 -0700, Tyrel Newton wrote:
>> I'm trying to use scipy.io.savemat to export a very large set of data to
>> a .mat file. The dataset contains around 20 million floats. When I try
>> to export this to a .mat file, I get a MemoryError. The specific
>> MemoryError is:
>> File "C:\Python26\lib\site-packages\scipy\io\matlab\miobase.py", line
>> 557 in write_bytes
> What is the complete error message? -- It typically indicates the
> specific part in C code the error originates from. (The full traceback,
Attached is a Windows command line screenshot my colleague captured.
> On the other hand, along that code path it seems the only source of a
> MemoryError can really be a failure to allocate memory for the tostring.
> Your data apparently needs 160 MB free for this to succeed -- which is
> not so much. So the question comes to what is the memory usage of the
> code when saving, compared to the available free memory?
Yeah, my theory is that the tostring process is basically trying to duplicate the memory usage by creating a string that is then written to the file. This seems like an inefficient way to do it, but my understanding of the code is limited, so I'm probably missing something.
-------------- next part --------------
A non-text attachment was scrubbed...
Size: 108251 bytes
Desc: not available
Url : http://mail.scipy.org/pipermail/scipy-user/attachments/20100725/d72276a5/attachment-0001.png
More information about the SciPy-User