[Numpy-discussion] nditer: possible to manually handle dimensions with different lengths?
John Salvatier
jsalvati@u.washington....
Mon Oct 3 16:03:02 CDT 2011
Some observations and questions about nested_iters. Nested_iters seems to
require that all input arrays have the same number of dimensions (so you
will have to pad some input shapes with 1s). Is there a way to specify how
the axes line are matched together like for nditer?
When I try to run the following program,
@cython.boundscheck(False)
def vars(vals, group, axis ):
cdef np.ndarray[double, ndim = 2] values
cdef np.ndarray[long long, ndim = 2] groups
cdef np.ndarray[double, ndim = 2] outs
cdef int size
cdef double value
cdef int i, j
cdef long long cgroup
cdef double min
cdef double max
cdef double open
oshape = list(vals.shape)
bins = len(np.unique(group))
oshape = oshape+[bins]
oshape[axis] = 1
out = np.empty(tuple(oshape))
axes = range(vals.ndim)
axes.remove(axis)
gshape = [1] * len(oshape)
gshape[axis] = len(group)
group.shape = gshape
vals = vals[...,np.newaxis]
it0, it1 = np.nested_iters([vals,group, out],
[axes, [axis,len(oshape) -1]],
op_dtypes=['float64', 'int64', 'float64'],
flags = ['multi_index', 'buffered'])
size = vals.shape[axis]
for x in it0:
values, groups, outs = it1.itviews
j = -1
for i in range(size):
if cgroup != groups[i,0]:
if j != -1:
outs[0,j] = garmanklass(open, values[i,0], min, max)
cgroup = groups[i,0]
min = inf
max = -inf
open = values[i,0]
j += 1
min = fmin(min, values[i,0])
max = fmax(max, values[i,0])
outs[0,j+1] = garmanklass(open, values[size -1], min, max)
return out
I get an error
File "comp.pyx", line 58, in varscale.comp.vars (varscale\comp.c:1565)
values, groups, outs = it1.itviews
ValueError: cannot provide an iterator view when buffering is enabled
Which I am not sure how to deal with. Any advice?
What I am trying to do here is to do a "grouped" calculation (the group
specified by the group argument) on the values along the given axis. I try
to use nested_iter to iterate over the specified axis and a new axis (the
length of the number of groups) separately so I can do my calculation.
On Mon, Oct 3, 2011 at 9:03 AM, John Salvatier <jsalvati@u.washington.edu>wrote:
> Thanks mark! I think that's exactly what I'm looking for. We even had a
> previous discussion about this (oops!) (
> http://mail.scipy.org/pipermail/numpy-discussion/2011-January/054421.html
> ).
>
> I didn't find any documentation, I will try to add some once I understand
> how it works better.
>
> John
>
>
> On Sat, Oct 1, 2011 at 2:53 PM, Mark Wiebe <mwwiebe@gmail.com> wrote:
>
>> On Sat, Oct 1, 2011 at 1:45 PM, John Salvatier <jsalvati@u.washington.edu
>> > wrote:
>>
>>> I apologize, I picked a poor example of what I want to do. Your
>>> suggestion would work for the example I provided, but not for a more complex
>>> example. My actual task is something like a "group by" operation along a
>>> particular axis (with a known number of groups).
>>>
>>> Let me try again: What I would like to be able to do is to specify some
>>> of the iterator dimensions to be handled manually by me. For example lets
>>> say I have some kind of a 2d smoothing algorithm. If I start with an array
>>> of shape [a,b,c,d] and I'd like to do the 2d smoothing over the 2nd and 3rd
>>> dimensions, I'd like to be able to tell nditer to do normal broadcasting and
>>> iteration over the 1st and 4th dimensions but leave iteration over the 2nd
>>> and 3rd dimensions to me and my algorithm. Each iteration of nditer would
>>> give me a 2d array to which I apply my algorithm. This way I could write
>>> more arbitrary functions that operate on arrays and support broadcasting.
>>>
>>> Is clearer?
>>>
>>
>> Maybe this will work for you:
>>
>> In [15]: a = np.arange(2*3*4*5).reshape(2,3,4,5)
>>
>> In [16]: it0, it1 = np.nested_iters(a, [[0,3], [1,2]],
>> flags=['multi_index'])
>>
>> In [17]: for x in it0:
>> ....: print it1.itviews[0]
>> ....:
>> [[ 0 5 10 15]
>> [20 25 30 35]
>> [40 45 50 55]]
>> [[ 1 6 11 16]
>> [21 26 31 36]
>> [41 46 51 56]]
>> [[ 2 7 12 17]
>> [22 27 32 37]
>> [42 47 52 57]]
>> [[ 3 8 13 18]
>> [23 28 33 38]
>> [43 48 53 58]]
>> [[ 4 9 14 19]
>> [24 29 34 39]
>> [44 49 54 59]]
>> [[ 60 65 70 75]
>> [ 80 85 90 95]
>> [100 105 110 115]]
>> [[ 61 66 71 76]
>> [ 81 86 91 96]
>> [101 106 111 116]]
>> [[ 62 67 72 77]
>> [ 82 87 92 97]
>> [102 107 112 117]]
>> [[ 63 68 73 78]
>> [ 83 88 93 98]
>> [103 108 113 118]]
>> [[ 64 69 74 79]
>> [ 84 89 94 99]
>> [104 109 114 119]]
>>
>> Cheers,
>> Mark
>>
>>
>>
>>
>>>
>>>
>>> On Fri, Sep 30, 2011 at 5:04 PM, Mark Wiebe <mwwiebe@gmail.com> wrote:
>>>
>>>> On Fri, Sep 30, 2011 at 8:03 AM, John Salvatier <
>>>> jsalvati@u.washington.edu> wrote:
>>>>
>>>>> Using nditer, is it possible to manually handle dimensions with
>>>>> different lengths?
>>>>>
>>>>> For example, lets say I had an array A[5, 100] and I wanted to sample
>>>>> every 10 along the second axis so I would end up with an array B[5,10]. Is
>>>>> it possible to do this with nditer, handling the iteration over the second
>>>>> axis manually of course (probably in cython)?
>>>>>
>>>>> I want something like this (modified from
>>>>> http://docs.scipy.org/doc/numpy/reference/arrays.nditer.html#putting-the-inner-loop-in-cython
>>>>> )
>>>>>
>>>>> @cython.boundscheck(False)
>>>>> def sum_squares_cy(arr):
>>>>> cdef np.ndarray[double] x
>>>>> cdef np.ndarray[double] y
>>>>> cdef int size
>>>>> cdef double value
>>>>> cdef int j
>>>>>
>>>>> axeslist = list(arr.shape)
>>>>> axeslist[1] = -1
>>>>>
>>>>> out = zeros((arr.shape[0], 10))
>>>>> it = np.nditer([arr, out], flags=['reduce_ok', 'external_loop',
>>>>> 'buffered', 'delay_bufalloc'],
>>>>> op_flags=[['readonly'], ['readwrite', 'no_broadcast']],
>>>>> op_axes=[None, axeslist],
>>>>> op_dtypes=['float64', 'float64'])
>>>>> it.operands[1][...] = 0
>>>>> it.reset()
>>>>> for xarr, yarr in it:
>>>>> x = xarr
>>>>> y = yarr
>>>>> size = x.shape[0]
>>>>> j = 0
>>>>> for i in range(size):
>>>>> #some magic here involving indexing into x[i] and y[j]
>>>>> return it.operands[1]
>>>>>
>>>>> Does this make sense? Is it possible to do?
>>>>>
>>>>
>>>> I'm not sure I understand precisely what you're asking. Maybe you could
>>>> reshape A to have shape [5, 10, 10], so that one of those 10's can match up
>>>> with the 10 in B, perhaps with the op_axes?
>>>>
>>>> -Mark
>>>>
>>>>
>>>>>
>>>>> _______________________________________________
>>>>> NumPy-Discussion mailing list
>>>>> NumPy-Discussion@scipy.org
>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> NumPy-Discussion mailing list
>>>> NumPy-Discussion@scipy.org
>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>>
>>>>
>>>
>>> _______________________________________________
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion@scipy.org
>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>
>>>
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/numpy-discussion/attachments/20111003/c5a9027b/attachment-0001.html
More information about the NumPy-Discussion
mailing list