[SciPy-User] memory errors when using savemat
Tyrel Newton
newton@tethers....
Sun Jul 25 11:32:17 CDT 2010
> Could you give us a small script which reproduces the error ? 20
> million is pretty small, I suspect something else or a bug in scipy,
Not really, but I think I can list off the important parts.
1) Once I know the length, I declare an empty array with:
dataarr = numpy.empty(length, dtype=numpy.dtype(float))
2) I then populate the array with values, potentially out-of-order (which is I was a declare a large empty array first). The arrays are populated with something like:
dataarr[index] = float(datavalue)
3) I then create a dict object with multiple of these large float-type arrays, usually less than 6 arrays total. Each array is the same length, in this case 20M samples.
4) To this dict object, I add a few single-entry float arrays and strings that represent metadata for the large arrays. These are used to automatically process the data in MATLAB. Examples of creating the numpy types:
dataarr = numpy.empty(1, dtype=numpy.dtype(float))
dataarr[0] = datavalue
strdata = numpy.core.defchararray.array(str)
5) I then export the entire dict object in a single call:
scipy.io.savemat(fname, datadict)
Hope that's enough to explain what's going on.
Thanks,
Tyrel
More information about the SciPy-User
mailing list