[SciPy-User] Strange memory limits

Chris Weisiger cweisiger@msg.ucsf....
Tue Mar 29 12:31:23 CDT 2011


Thanks for the information, all. Annoyingly, I can't allocate 3 individual
735x512x512 arrays (each ~360MB) -- the third gives a memory error. If I
allocate 1024x1024 byte arrays (thus, each 1MB), I can make 1500 before
getting a memory error. So I'm definitely running into *some* issue that
prevents larger blocks from being allocated, but I'm also hitting a ceiling
well before I should be.

I had thought that Windows allowed for 3GB address spaces for 32-bit
processes, but apparently (per
http://msdn.microsoft.com/en-us/library/aa366778%28v=vs.85%29.aspx#memory_limits)
that only applies if the program has IMAGE_FILE_LARGE_ADDRESS_AWARE
and
4GT set...sounds like I'd need a recompile and some system tweaks to set
those. The proper, and more work-intensive, solution would be to make a
64-bit build.

My (admittedly limited) understanding of memory fragmentation was that it's
a per-process problem. I'm seeing this issue immediately on starting up the
program, so the program's virtual memory address space should be pretty
clean.

-Chris

On Tue, Mar 29, 2011 at 10:14 AM, Christopher Barker
<Chris.Barker@noaa.gov>wrote:

> On 3/28/11 4:51 PM, David Baddeley wrote:
> > what you're probably running into is a problem with allocating a
> > continuous block of memory / a memory fragmentation issue.
>
> On 3/28/11 8:01 PM, Charles R Harris wrote:
>
> > Windows 32 bit gives you 2GB and keeps the rest for itself.
>
> right -- I've had no problems running the same code with 32 bit python
> on OS-X that crashes out with memory errors on Windows -- similar hardware.
>
> > Compiling as 64 bit might solve your problem, as, with 12 GB of memory,
> > there will be a larger address space to look for contiguous blocks in,
> > but probably doesn't address the fundamental issue.
>
> Ah, but while you still may only have 12GB memory, with 64 bit Windows
> and Python, the virtual memory space is massive, so I suspect you'll be
> fine. Using 1GB memory buffers on 32bit is certainly pushing it.
>
> > I suspect you could
> > probably get away with having much smaller contiguous blocks (eg have 3
> > separate arrays for the 3 different cameras) or even a new array for
> > each image.
>
> That would make it easier for the OS to manage the memory well.
>
> -Chris
>
>
> --
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> Chris.Barker@noaa.gov
> _______________________________________________
> SciPy-User mailing list
> SciPy-User@scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/scipy-user/attachments/20110329/850c1c94/attachment.html 


More information about the SciPy-User mailing list