[Numpy-discussion] Huge arrays
Charles R Harris
Tue Sep 8 20:41:23 CDT 2009
On Tue, Sep 8, 2009 at 7:30 PM, Daniel Platz <
> I have a numpy newbie question. I want to store a huge amount of data
> in an array. This data come from a measurement setup and I want to
> write them to disk later since there is nearly no time for this during
> the measurement. To put some numbers up: I have 2*256*2000000 int16
> numbers which I want to store. I tried
> data1 = numpy.zeros((256,2000000),dtype=int16)
> data2 = numpy.zeros((256,2000000),dtype=int16)
> This works for the first array data1. However, it returns with a
> memory error for array data2. I have read somewhere that there is a
> 2GB limit for numpy arrays on a 32 bit machine but shouldn't I still
> be below that? I use Windows XP Pro 32 bit with 3GB of RAM.
More precisely, 2GB for windows and 3GB for (non-PAE enabled) linux. The
rest of the address space is set aside for the operating system. Note that
address space is not the same as physical memory, but it sets a limit on
what you can use, whether swap or real memory.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the NumPy-Discussion