[Numpy-discussion] array slicing questions

Nathaniel Smith njs@pobox....
Tue Jul 31 11:30:15 CDT 2012


On Tue, Jul 31, 2012 at 4:57 PM, eat <e.antero.tammi@gmail.com> wrote:
> Hi,
>
> On Tue, Jul 31, 2012 at 6:43 PM, Nathaniel Smith <njs@pobox.com> wrote:
>>
>> On Tue, Jul 31, 2012 at 2:23 PM, eat <e.antero.tammi@gmail.com> wrote:
>> > Apparently ast(.) does not return a view of the original matches rather
>> > a
>> > copy of size (n* (2* distance+ 1)), thus you may run out of memory.
>>
>> The problem isn't memory, it's that on 32-bit Python,
>> np.prod(arr.shape) must be <2**32 (or maybe 2**31 -- something like
>> that).
>
> I think this is what the traceback is indicating.
>>
>> Normally you wouldn't be creating such arrays anyway because
>> they would be too big to fit into memory, so this problem isn't
>> observed, but when you're using stride_tricks then it's very easy to
>> create arrays that use only a small amount of memory but that have
>> very large shapes.
>
> But in this specific case .nbytes attribute indicates that a huge amount of
> memory is used. So I guess stride_tricks(.) is not returning a view.

No, .nbytes is lying to you -- it just returns np.prod(arr.shape) *
arr.dtype.itemsize. It isn't smart enough to realize that you have
wacky strides that cause the same memory region to be referenced by
many different array elements.

-n


More information about the NumPy-Discussion mailing list