[Numpy-discussion] memory error caused by astype()

Gong, Shawn (Contractor) Shawn.Gong@drdc-rddc.gc...
Wed Aug 15 09:47:08 CDT 2007


Hi list,

When I do large array manipulations, I get out-of-memory errors.
If the array size is 5000 by 6000, the following codes use nearly 1G.
Then my PC displays a Python error box.  The try/except won't catch it
if the memory error happens in "astype" instead of "array1* array2"

    try:
        if ( array1.typecode() in cplx_types ):
            array1 = abs(array1.astype(Numeric.Complex32))
        else:
            array1 = array1.astype(Numeric.Float32)

        if ( array2.typecode() in cplx_types ):
            array2 = abs(array2.astype(Numeric.Complex32))
        else:
            array2 = array2.astype(Numeric.Float32)

        array1 = Numeric.sqrt(array1) * Numeric.sqrt(array2)
        return array1

    except:
        gvutils.error("Memory error occurred\nPlease select a smaller
array")
        return None

My questions are:
1)	Is there a more memory efficient way of doing this?
2)	How do I deal with exception if astype is the only way to go
3)	Is there a way in Python that detects the available RAM and
limits the array size before he/she can go ahead with the array
multiplications?  
			i.e. detects the available RAM, say 800K 
			Assume worst case - Complex32
			Figure out how many temp_arrays used by numpy
			Calculate array size limit = ??
4)	If there is no 3) Is there something in Python that monitors
memory and warns the user. I have these "astype" at a number functions.
Do I have to put try/except at each location?

Thanks,
Shaw Gong


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://projects.scipy.org/pipermail/numpy-discussion/attachments/20070815/e876c448/attachment.html 


More information about the Numpy-discussion mailing list