[Numpy-discussion] Huge arrays

Sebastian Haase seb.haase@gmail....
Wed Sep 9 00:10:58 CDT 2009


Hi,
you can probably use PyTables for this. Even though it's meant to
save/load data to/from disk (in HDF5 format) as far as I understand,
it can be used to make your task solvable - even on a 32bit system !!
It's free (pytables.org) -- so maybe you can try it out and tell me if
I'm right ....
Or someone else here would know right away...

Cheers,
Sebastian Haase


On Wed, Sep 9, 2009 at 6:19 AM, Sturla Molden<sturla@molden.no> wrote:
> Daniel Platz skrev:
>> data1 = numpy.zeros((256,2000000),dtype=int16)
>> data2 = numpy.zeros((256,2000000),dtype=int16)
>>
>> This works for the first array data1. However, it returns with a
>> memory error for array data2. I have read somewhere that there is a
>> 2GB limit for numpy arrays on a 32 bit machine but shouldn't I still
>> be below that? I use Windows XP Pro 32 bit with 3GB of RAM.
>
> There is a 2 GB limit for user space on Win32, this is about 1.9 GB. You
> have other programs running as well, so this is still too much. Also
> Windows reserves 50% of RAM for itself, so you have less than 1.5 GB to
> play with.
>
> S.M.
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>


More information about the NumPy-Discussion mailing list