[SciPy-User] MemoryError with tsfromtxt

Pierre GM pgmdevlist@gmail....
Thu Sep 9 06:19:53 CDT 2010

On Sep 9, 2010, at 1:11 PM, Timmie wrote:

>> You must have quite a huge file... Note that it's not a scikits.timeseries 
> pb, just a standard numpy one.
> The file has 298 MB.
> 5370772 records (rows); data in minutely frequency.
>>> And what could I do to mitigate it?
>> Cut the file in pieces ?
> and then concatenate the timeseries? 

That's the idea. Could you cut it day by day, or week by week, or even month by month to reduce the load ? 
The issue is that genfromtxt has to keep a lot of information in memory (a list of values, a list of masks) before creating the array, and you're overloading Python's capacity to deal with it...

More information about the SciPy-User mailing list