[SciPy-user] NaN's in numpy (and Scipy)
Alan Jackson
alan at ajackson.org
Sat Jan 6 18:22:42 CST 2007
On Sat, 6 Jan 2007 16:25:56 -0500
Pierre GM <pgmdevlist at gmail.com> wrote:
> On Saturday 06 January 2007 16:08, Alan Jackson wrote:
> > Missing data. Basically want to handle missing data as painlessly as
> > possible without having to add a bunch of stuff every time a calculation is
> > done.
>
> Alan,
> Could you be a bit more specific about you mean by this "bunch of stuffs" ?
> What are your complaints about the current implementation of masked arrays ?
Well for one it seems a shame to carry around the extra storage when 99.9%
of my data is not missing.
It may be my own inexperience with python, numpy, and masked arrays, but it
seemed more complex than necessary to do the following. I wanted to take an
array and find the forward (or is it backward? one-sided anyway) difference,
and then append the final element so the output array had the same size as the
input. With normal arrays this could be done with
b = concatenate((a[1:]-a[:-1],[a[-1]-a[-2]] )) or something close to that, I'm
writing from memory. With masked arrays, you seem to lose the 'maskedness'
when they are only one-d, so the concatenate fails and you have to fiddle about
a bit to get it to work right. In my mind, it should "just work", so that the
code looks the same in either case, but I couldn't get it to do that.
Perhaps the new implementation has taken care of that. If so, that would take
care of a lot of my pain. I'm still pushing my IT support guys to install the
latest release.
Hmmm... just tried it and it seems to work. I definitely need to start using
a newer version at work. I really hate that my home system is so much more up
to date. I'll get back to you!
Thanks!
> Assuming you have Nan in your data, you can get a masked array as easily as
> that:
> >>> import numpy as N
> >>> import numpy.core.ma as MA
> >>> x = N.array([1,2,N.nan,4])
> >>> X = MA.masked_array(x, mask=N.isnan(x)
> >>> X
> array(data =
> [ 1.00000000e+00 2.00000000e+00 1.00000000e+20 4.00000000e+00],
> mask =
> [False False True False],
> fill_value=1e+20)
>
> And then you can play with X.
>
> A few months ago, I ran into some problems while trying to subclass
> MaskedArray. I rewrote most of numpy.core.ma to solve my particular issues.
> This new implementation is available on the scipy SVN server, in the
> sandbox/maskedarray folder. I'd be glad if you could give it a try, so that I
> could improve it.
> _______________________________________________
> SciPy-user mailing list
> SciPy-user at scipy.org
> http://projects.scipy.org/mailman/listinfo/scipy-user
--
-----------------------------------------------------------------------
| Alan K. Jackson | To see a World in a Grain of Sand |
| alan at ajackson.org | And a Heaven in a Wild Flower, |
| www.ajackson.org | Hold Infinity in the palm of your hand |
| Houston, Texas | And Eternity in an hour. - Blake |
-----------------------------------------------------------------------
More information about the SciPy-user
mailing list