[Numpy-discussion] A memory problem: why does mmap come up in numpy.inner?
Charles R Harris
charlesr.harris@gmail....
Wed Jun 4 22:26:44 CDT 2008
On Wed, Jun 4, 2008 at 9:07 PM, Anne Archibald <peridot.faceted@gmail.com>
wrote:
> 2008/6/4 Dan Yamins <dyamins@gmail.com>:
> >
> >
> >
> >>
> >> Try
> >>
> >> In [3]: numpy.dtype(numpy.uintp).itemsize
> >> Out[3]: 4
> >>
> >> which is the size in bytes of the integer needed to hold a pointer. The
> >> output above is for 32 bit python/numpy.
> >>
> >> Chuck
> >
> > Check, the answer is 4, as you got for the 32-bit. What would the
> answer
> > be on a 64-bit architecture? Why is this diagnostic?
>
> In a 64-bit setting, a pointer needs to be 64 bits long, that is,
> eight bytes, not four.
>
> What Charles pointed out was that while the inner product is very big,
> it seems to fit into memory on his 32-bit Linux machine; is it
> possible that OSX is preventing your python process from using even
> the meager 2-3 GB that a 32-bit process ought to get? In particular,
> try running Charles' script in a fresh python interpreter and see if
> it works; it may be that other arrays you had allocated are taking up
> some of the space that this one could.
>
> You will probably still want a 64-bit python, though, in order to have
> a little elbow room.
>
I think the difference is that Linux takes up 1 GiB, leaving 3 GiB to the
process, while I suspect OS X is taking up 2 GiB. I don't have a Mac, so I
don't really know. When I ran the the problem it took about 1.8 GiB, making
it a close run thing if the Mac only gives 32 bit proccesses 2 GiB.
Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://projects.scipy.org/pipermail/numpy-discussion/attachments/20080604/3b24b3bd/attachment.html
More information about the Numpy-discussion
mailing list