[SciPy-User] memory errors when using savemat

Tyrel Newton newton@tethers....
Sun Jul 25 11:32:17 CDT 2010


> Could you give us a small script which reproduces the error ? 20
> million is pretty small, I suspect something else or a bug in scipy,

Not really, but I think I can list off the important parts.

1) Once I know the length, I declare an empty array with:
	dataarr = numpy.empty(length, dtype=numpy.dtype(float))

2) I then populate the array with values, potentially out-of-order (which is I was a declare a large empty array first). The arrays are populated with something like:
	dataarr[index] = float(datavalue)

3) I then create a dict object with multiple of these large float-type arrays, usually less than 6 arrays total. Each array is the same length, in this case 20M samples.

4) To this dict object, I add a few single-entry float arrays and strings that represent metadata for the large arrays. These are used to automatically process the data in MATLAB. Examples of creating the numpy types:
	dataarr = numpy.empty(1, dtype=numpy.dtype(float))
	dataarr[0] = datavalue
	strdata = numpy.core.defchararray.array(str)

5) I then export the entire dict object in a single call:
	scipy.io.savemat(fname, datadict)

Hope that's enough to explain what's going on.

Thanks,
Tyrel




More information about the SciPy-User mailing list