[Numpy-discussion] advanced indexing bug with huge arrays?
Robin
robince@gmail....
Mon Jan 23 13:38:44 CST 2012
On Mon, Jan 23, 2012 at 7:55 PM, David Warde-Farley
<wardefar@iro.umontreal.ca> wrote:
> I've reproduced this (rather serious) bug myself and confirmed that it exists
> in master, and as far back as 1.4.1.
>
> I'd really appreciate if someone could reproduce and confirm on another
> machine, as so far all my testing has been on our single high-memory machine.
I see the same behaviour on a Winodows machine with numpy 1.6.1. But I
don't think it is an indexing problem - rather something with the
random number creation. a itself is already zeros for high indexes.
In [8]: b[1000000:1000010]
Out[8]:
array([3429029, 1251819, 4292918, 2249483, 757620, 3977130, 3455449,
2005054, 2565207, 3114930])
In [9]: a[b[1000000:1000010]]
Out[9]:
array([[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]], dtype=uint8)
In [41]: a[581350:,0].sum()
Out[41]: 0
Cheers
Robin
>
> Thanks,
> David
>
> On Mon, Jan 23, 2012 at 05:23:28AM -0500, David Warde-Farley wrote:
>> A colleague has run into this weird behaviour with NumPy 1.6.1, EPD 7.1-2, on Linux (Fedora Core 14) 64-bit:
>>
>> > a = numpy.array(numpy.random.randint(256,size=(5000000,972)),dtype='uint8')
>> > b = numpy.random.randint(5000000,size=(4993210,))
>> > c = a[b]
>>
>> It seems c is not getting filled in full, namely:
>>
>> > In [14]: c[1000000:].sum()
>> > Out[14]: 0
>>
>> I haven't been able to reproduce this quite yet, I'll try to find a machine with sufficient memory tomorrow. But does anyone have any insight in the mean time? It smells like some kind of integer overflow bug.
>>
>> Thanks,
>>
>> David
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
More information about the NumPy-Discussion
mailing list