[SciPy-user] NaN's in numpy (and Scipy)

Alan Jackson alan at ajackson.org
Sat Jan 6 18:22:42 CST 2007


On Sat, 6 Jan 2007 16:25:56 -0500
Pierre GM <pgmdevlist at gmail.com> wrote:

> On Saturday 06 January 2007 16:08, Alan Jackson wrote:
> > Missing data. Basically want to handle missing data as painlessly as
> > possible without having to add a bunch of stuff every time a calculation is
> > done.
> 
> Alan,
> Could you be a bit more specific about you mean by this "bunch of stuffs" ? 
> What are your complaints about the current implementation of masked arrays ?

Well for one it seems a shame to carry around the extra storage when 99.9% 
of my data is not missing.

It may be my own inexperience with python, numpy, and masked arrays, but it
seemed more complex than necessary to do the following. I wanted to take an
array and find the forward (or is it backward? one-sided anyway) difference,
and then append the final element so the output array had the same size as the
input. With normal arrays this could be done with 
b = concatenate((a[1:]-a[:-1],[a[-1]-a[-2]] )) or something close to that, I'm
writing from memory.  With masked arrays, you seem to lose the 'maskedness'
when they are only one-d, so the concatenate fails and you have to fiddle about
a bit to get it to work right. In my mind, it should "just work", so that the
code looks the same in either case, but I couldn't get it to do that.

Perhaps the new implementation has taken care of that. If so, that would take 
care of a lot of my pain. I'm still pushing my IT support guys to install the 
latest release.

Hmmm... just tried it and it seems to work. I definitely need to start using 
a newer version at work. I really hate that my home system is so much more up 
to date. I'll get back to you!


Thanks!


> Assuming you have Nan in your data, you can get a masked array as easily as
> that:
> >>> import numpy as N
> >>> import numpy.core.ma as MA
> >>> x = N.array([1,2,N.nan,4])
> >>> X = MA.masked_array(x, mask=N.isnan(x)
> >>> X
> array(data = 
>  [  1.00000000e+00   2.00000000e+00   1.00000000e+20   4.00000000e+00],
>       mask =
>  [False False True False],
>       fill_value=1e+20)
> 
> And then you can play with X.
> 
> A few months ago, I ran into some problems while trying to subclass 
> MaskedArray. I rewrote most of numpy.core.ma to solve my particular issues.
> This new implementation is available on the scipy SVN server, in the 
> sandbox/maskedarray folder. I'd be glad if you could give it a try, so that I 
> could improve it.
> _______________________________________________
> SciPy-user mailing list
> SciPy-user at scipy.org
> http://projects.scipy.org/mailman/listinfo/scipy-user


-- 
-----------------------------------------------------------------------
| Alan K. Jackson            | To see a World in a Grain of Sand      |
| alan at ajackson.org          | And a Heaven in a Wild Flower,         |
| www.ajackson.org           | Hold Infinity in the palm of your hand |
| Houston, Texas             | And Eternity in an hour. - Blake       |
-----------------------------------------------------------------------


More information about the SciPy-user mailing list