[Numpy-discussion] Are masked arrays slower for processing than ndarrays?
Wed May 13 17:35:21 CDT 2009
I just committed (r6994) some modifications to numpy.ma.getdata (Eric
Firing's patch) and to the ufunc wrappers that were too slow with
large arrays. We're roughly 3 times faster than we used to, but still
slower than the equivalent classic ufuncs (no surprise here).
Here's the catch: it's basically cheating. I got rid of the pre-
processing (where a mask was calculated depending on the domain and
the input set to a filling value depending on this mask, before the
actual computation). Instead, I force
np.seterr(divide='ignore',invalid='ignore') before calling the ufunc
on the .data part, then mask the invalid values (if any) and reset the
corresponding entries in .data to the input. Finally, I reset the
error status. All in all, we're still data-friendly, meaning that the
value below a masked entry is the same as the input, but we can't say
that values initially masked are discarded (they're used in the
computation but reset to their initial value)...
This playing around with the error status may (or may not, I don't
know) cause some problems down the road.
It's still faaar faster than computing the domain (especially
_DomainSafeDivide) when the inputs are large...
I'd be happy if you could give it a try and send some feedback.
On May 9, 2009, at 8:17 PM, Eric Firing wrote:
> Eric Firing wrote:
> ... I pressed "send" too soon. There are test failures with the
> patch I attached to my last message. I think the basic ideas are
> correct, but evidently there are wrinkles to be worked out. Maybe
> putmask() has to be used instead of where() (putmask is much faster)
> to maintain the ability to do *= and similar, and maybe there are
> other adjustments. Somehow, though, it should be possible to get
> decent speed for simple multiplication and division; a 10x penalty
> relative to ndarray operations is just too much.
>> Eli Bressert wrote:
>>> I'm using masked arrays to compute large-scale standard deviation,
>>> multiplication, gaussian, and weighted averages. At first I thought
>>> using the masked arrays would be a great way to sidestep looping
>>> (which it is), but it's still slower than expected. Here's a snippet
>>> of the code that I'm using it for.
>>> # Like the spatial_weight section, this takes about 20 seconds
>>> W = spatial_weight / Rho2
>>> # Takes less than one second.
>>> Ave = np.average(av_good,axis=1,weights=W)
>>> Any ideas on why it would take such a long time for processing?
>> A part of the slowdown is what looks to me like unnecessary copying
>> in _MaskedBinaryOperation.__call__. It is using getdata, which
>> applies numpy.array to its input, forcing a copy. I think the copy
>> is actually unintentional, in at least one sense, and possibly two:
>> first, because the default argument of getattr is always evaluated,
>> even if it is not needed; and second, because the call to np.array
>> is used where np.asarray or equivalent would suffice.
>> The first file attached below shows the kernprof in the case of
>> multiplying two masked arrays, shape (100000,50), with no masked
>> elements; 2/3 of the time is taken copying the data.
>> Now, if there are actually masked elements in the arrays, it gets
>> much worse: see the second attachment. The total time has
>> increased by more than a factor of 3, and the culprit is
>> numpy.which(), a very slow function. It looks to me like it is
>> doing nothing useful at all; the numpy binary operation is still
>> being executed for all elements, regardless of mask, contrary to
>> the intention implied by the comment in the code.
>> The third attached file has a patch that fixes the getdata problem
>> and eliminates the which().
>> With this patch applied we get the profile in the 4th file, to be
>> compared to the second profile. Much better. I am pretty sure it
>> could still be sped up quite a bit, though. It looks like the
>> masks are essentially being calculated twice for no good reason,
>> but I don't completely understand all the mask considerations, so
>> at this point I am not trying to fix that problem.
>>> Especially the spatial_weight and W variables? Would there be a
>>> way to do this? Or is there a way that numpy.std can process ignore
>>> nan's when processing?
>>> Eli Bressert
>>> Numpy-discussion mailing list
>> Numpy-discussion mailing list
More information about the Numpy-discussion