[Numpy-discussion] Huge arrays
Tue Sep 8 19:53:01 CDT 2009
On Wed, Sep 9, 2009 at 9:30 AM, Daniel
> I have a numpy newbie question. I want to store a huge amount of data
> in an array. This data come from a measurement setup and I want to
> write them to disk later since there is nearly no time for this during
> the measurement. To put some numbers up: I have 2*256*2000000 int16
> numbers which I want to store. I tried
> data1 = numpy.zeros((256,2000000),dtype=int16)
> data2 = numpy.zeros((256,2000000),dtype=int16)
> This works for the first array data1. However, it returns with a
> memory error for array data2. I have read somewhere that there is a
> 2GB limit for numpy arrays on a 32 bit machine
This has nothing to do with numpy per se - that's the fundamental
limitation of 32 bits architectures. Each of your array is 1024 Mb, so
you won't be able to create two of them.
The 2Gb limit is a theoretical upper limit, and in practice, it will
always be lower, if only because python itself needs some memory.
There is also the memory fragmentation problem, which means allocating
one contiguous, almost 2Gb segment will be difficult.
> If someone has an idea to help me I would be very glad.
If you really need to deal with arrays that big, you should move on 64
bits architecture. That's exactly the problem they are solving.
More information about the NumPy-Discussion