[Numpy-discussion] A memory problem: why does mmap come up in numpy.inner?

Charles R Harris charlesr.harris@gmail....
Wed Jun 4 20:59:40 CDT 2008


On Wed, Jun 4, 2008 at 7:41 PM, Dan Yamins <dyamins@gmail.com> wrote:

>
>
> On Wed, Jun 4, 2008 at 9:06 PM, Charles R Harris <
> charlesr.harris@gmail.com> wrote:
>
>>
>>
>> On Wed, Jun 4, 2008 at 6:42 PM, Dan Yamins <dyamins@gmail.com> wrote:
>>
>>> I'm using python 2.5.2 on OS X, with 8 GB of ram, and a 64-bit
>>> processor.  In
>>> this, setting, I'm working with large arrays of binary data.  E.g, I want
>>> to
>>> make calls like:
>>>                Z = numpy.inner(a,b)
>>> where and b are fairly large  -- e.g. 20000 rows by 100 columns.
>>>
>>> However, when such a call is made, I get a memory error that I don't
>>> understand.
>>> Specifically:
>>>
>>> >>> s = numpy.random.binomial(1,.5,(20000,100))   #creates 20000x100 bin.
>>> array
>>> >>> r = numpy.inner(s,s)
>>> Python(1714) malloc: *** mmap(size=1600000000) failed (error code=12)
>>> *** error: can't allocate region
>>> *** set a breakpoint in malloc_error_break to debug
>>>
>>>
>>>
>> Are both python and your version of OS X fully 64 bits?
>>
>
>
> I'm not sure.   My version of OS X is the most recent version, the one that
> ships with a new MacPro Dual Quad-core Xeon 3.2MHz chipset.  The processor
> is definitely 64-bit, so I think the operating system probably is enable for
> that, but am not sure. (How would I find out?)  As for the python version, I
> thought that 2.5 and above were 64-enabled, but I'm not sure how I'd check
> it.
>

Hmm,

In [1]: s = numpy.random.binomial(1,.5,(20000,100))

In [2]: inner(s,s)
Out[2]:
array([[45, 22, 17, ..., 20, 26, 23],
       [22, 52, 26, ..., 23, 33, 24],
       [17, 26, 52, ..., 27, 27, 19],
       ...,
       [20, 23, 27, ..., 46, 26, 22],
       [26, 33, 27, ..., 26, 54, 25],
       [23, 24, 19, ..., 22, 25, 44]])

This on 32 bit fedora 8 with 2GiB of actual memory. It was slow and a couple
of hundred megs of something went into swap, but it did complete. So this
looks to me like an OS X problem. Are there any limitations on the user
memory sizes? There might be some system setting accounting for this.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://projects.scipy.org/pipermail/numpy-discussion/attachments/20080604/0c102cf2/attachment.html 


More information about the Numpy-discussion mailing list