[Numpy-discussion] A memory problem: why does mmap come up in numpy.inner?
Charles R Harris
Wed Jun 4 22:26:44 CDT 2008
On Wed, Jun 4, 2008 at 9:07 PM, Anne Archibald <firstname.lastname@example.org>
> 2008/6/4 Dan Yamins <email@example.com>:
> >> Try
> >> In : numpy.dtype(numpy.uintp).itemsize
> >> Out: 4
> >> which is the size in bytes of the integer needed to hold a pointer. The
> >> output above is for 32 bit python/numpy.
> >> Chuck
> > Check, the answer is 4, as you got for the 32-bit. What would the
> > be on a 64-bit architecture? Why is this diagnostic?
> In a 64-bit setting, a pointer needs to be 64 bits long, that is,
> eight bytes, not four.
> What Charles pointed out was that while the inner product is very big,
> it seems to fit into memory on his 32-bit Linux machine; is it
> possible that OSX is preventing your python process from using even
> the meager 2-3 GB that a 32-bit process ought to get? In particular,
> try running Charles' script in a fresh python interpreter and see if
> it works; it may be that other arrays you had allocated are taking up
> some of the space that this one could.
> You will probably still want a 64-bit python, though, in order to have
> a little elbow room.
I think the difference is that Linux takes up 1 GiB, leaving 3 GiB to the
process, while I suspect OS X is taking up 2 GiB. I don't have a Mac, so I
don't really know. When I ran the the problem it took about 1.8 GiB, making
it a close run thing if the Mac only gives 32 bit proccesses 2 GiB.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Numpy-discussion