[SciPy-User] MemoryError with tsfromtxt
Thu Sep 9 06:19:53 CDT 2010
On Sep 9, 2010, at 1:11 PM, Timmie wrote:
>> You must have quite a huge file... Note that it's not a scikits.timeseries
> pb, just a standard numpy one.
> The file has 298 MB.
> 5370772 records (rows); data in minutely frequency.
>>> And what could I do to mitigate it?
>> Cut the file in pieces ?
> and then concatenate the timeseries?
That's the idea. Could you cut it day by day, or week by week, or even month by month to reduce the load ?
The issue is that genfromtxt has to keep a lot of information in memory (a list of values, a list of masks) before creating the array, and you're overloading Python's capacity to deal with it...
More information about the SciPy-User