[AstroPy] Speed of numpy masked arrays

Neil Crighton neilcrighton@gmail....
Wed Mar 7 10:11:38 CST 2012


Ah sorry, my mistake. I hadn't realised the .mean() method was pretty
fast. numpy.ma.median() seems to be much slower than numpy.median(),
though:

In [35]: myimages = numpy.ma.MaskedArray(rand(5,100,100), rand(5, 100,
100) > .5)
In [36]: %timeit np.ma.median(myimages, axis=0)
1 loops, best of 3: 2.33 s per loop
In [37]: myimages = rand(5,100,100)
In [38]: %timeit np.median(myimages, axis=0)
10 loops, best of 3: 1.4 ms per loop

Cheers, Neil

On 7 March 2012 14:52, Perry Greenfield <perry@stsci.edu> wrote:
> Hi Neil,
>
> That's a bit misleading. The application of the masking operation is
> certainly done in C. The whole issue is how tightly integrated the masks are
> with native arrays. For large arrays, it's not likely a significant speed
> issue per se, but it is perhaps a memory issue (with consequent speed issues
> if memory caches are involved).
>
> Perry
>
>
> On Mar 7, 2012, at 8:45 AM, Neil Crighton wrote:
>
>> Hi Wolfgang,
>>
>> The problem is that masked arrays in numpy are not implemented in c,
>> so they are not c-speed.  There is a proposal to implement a masked
>> array in c
>> (https://github.com/numpy/numpy/blob/master/doc/neps/missing-data.rst)
>> but this hasn't been finalised.
>>
>> Cheers, Neil
>>
>>
>> On 7 March 2012 14:41, Wolfgang Kerzendorf <wkerzendorf@gmail.com> wrote:
>>>
>>> Hello Jose,
>>>
>>> What you're trying to do is already half implemented in numpy and called
>>> masked arrays:
>>>
>>> Let's say you want to stack images and you have put them together in a 3d
>>> cube (first axis being the images):
>>>
>>> myimages = numpy.ma.MaskedArray(rand(5,100,100),
>>> mask=zeros((5,100,100).astype(bool))
>>>
>>> let's do a simple sigma clipping algorithm:
>>>
>>> image_mean = mean(myimages)
>>> image_std = std(myimages)
>>>
>>> #now let's adjust the mask to mask the pixels that are more than 1 sigma
>>> out
>>>
>>> new_mask = abs(myimages - image_mean) > 1.
>>>
>>> myimages.mask = new_mask
>>>
>>> #if you do a mean now it will ignore the values where the mask = True
>>>
>>> mean(myimages, axis=0)
>>>
>>> This will be all near C-speeds (I guess a factor of 100 is easily in
>>> there).
>>> ----
>>>
>>> As other's have pointed out using i,j in a loop is a very bad idea for
>>> numpy
>>> arrays. Numpy arrays are much more easy to handle than in most other
>>> languages, there is a bit of getting used to required though. I recommend
>>> reviewing the different operations
>>> on http://www.scipy.org/Tentative_NumPy_Tutorial.
>>>
>>> Hope that helps,
>>>
>>>    Wolfgang
>>> On 2012-03-07, at 4:41 AM, Sergio Pascual wrote:
>>>
>>> Hello
>>>
>>> I have written a image combine implementation for the pipeline of
>>> EMIR[1], a near infrared instrument for the 10m GTC Telescope. It does
>>> scaling,
>>> weighting and masking. It's written as a C/C++ extension. It's part of
>>> a larger package, numina, used for other GTC instruments, but I may
>>> split combine if there is interest in it. The package, called numina,
>>> its under heavy development, but the combine part is fairly stable.
>>>
>>> You can see the code here https://guaix.fis.ucm.es/hg/numina/
>>>
>>> The combine part is the module numina.array.combine, fairly all the
>>> C/C++ code is in src/
>>>
>>> Regards, Sergio
>>>
>>>
>>> [1] http://www.gtc.iac.es/pages/instrumentacion/emir.php
>>> [2] http://www.gtc.iac.es/
>>>
>>>
>>>
>>> El día 6 de marzo de 2012 12:00, Jose Miguel Ibáñez <ppmime@gmail.com>
>>> escribió:
>>>
>>> Hello everyone,
>>>
>>>
>>> does anyone know of an implementation of the iraf.imcombine task in
>>>
>>> python+numpy ? (of course, not using pyraf.imcombine call)
>>>
>>>
>>>
>>> Thanks !
>>>
>>> Jose
>>>
>>> _______________________________________________
>>>
>>> AstroPy mailing list
>>>
>>> AstroPy@scipy.org
>>>
>>> http://mail.scipy.org/mailman/listinfo/astropy
>>>
>>>
>>>
>>>
>>> --
>>> Sergio Pascual     http://guaix.fis.ucm.es/~spr    +34 91 394 5018
>>> gpg fingerprint: 5203 B42D 86A0 5649 410A F4AC A35F D465 F263 BCCC
>>> Departamento de Astrofísica -- Universidad Complutense de Madrid (Spain)
>>> _______________________________________________
>>> AstroPy mailing list
>>> AstroPy@scipy.org
>>> http://mail.scipy.org/mailman/listinfo/astropy
>>>
>>>
>>>
>>> _______________________________________________
>>> AstroPy mailing list
>>> AstroPy@scipy.org
>>> http://mail.scipy.org/mailman/listinfo/astropy
>>>
>> _______________________________________________
>> AstroPy mailing list
>> AstroPy@scipy.org
>> http://mail.scipy.org/mailman/listinfo/astropy
>
>


More information about the AstroPy mailing list