[Numpy-discussion] advanced indexing bug with huge arrays?

Travis Oliphant teoliphant@gmail....
Mon Jan 23 13:33:42 CST 2012


Can you determine where the problem is, precisely.    In other words, can you verify that c is not getting filled in correctly? 

You are no doubt going to get overflow in the summation as you have a uint8 parameter.   But, having that overflow be exactly '0' would be surprising.  

Can you verify that a and b are getting created correctly?   Also, 'c' should be a 2-d array, can you verify that?  Can you take the sum along the -1 axis and the 0 axis separately: 

print a.shape
print b.shape
print c.shape

c[1000000:].sum(axis=0)
d = c[1000000:].sum(axis=-1)
print d[:100]
print d[-100:]



On Jan 23, 2012, at 12:55 PM, David Warde-Farley wrote:

> I've reproduced this (rather serious) bug myself and confirmed that it exists
> in master, and as far back as 1.4.1.
> 
> I'd really appreciate if someone could reproduce and confirm on another
> machine, as so far all my testing has been on our single high-memory machine.
> 
> Thanks,
> David
> 
> On Mon, Jan 23, 2012 at 05:23:28AM -0500, David Warde-Farley wrote:
>> A colleague has run into this weird behaviour with NumPy 1.6.1, EPD 7.1-2, on Linux (Fedora Core 14) 64-bit:
>> 
>>> a = numpy.array(numpy.random.randint(256,size=(5000000,972)),dtype='uint8')
>>> b = numpy.random.randint(5000000,size=(4993210,))
>>> c = a[b]
>> 
>> It seems c is not getting filled in full, namely:
>> 
>>> In [14]: c[1000000:].sum()
>>> Out[14]: 0
>> 
>> I haven't been able to reproduce this quite yet, I'll try to find a machine with sufficient memory tomorrow. But does anyone have any insight in the mean time? It smells like some kind of integer overflow bug.
>> 
>> Thanks,
>> 
>> David
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion



More information about the NumPy-Discussion mailing list