Tue Jul 20 18:12:31 CDT 2010
On Tue, Jul 20, 2010 at 12:24 PM, <PHobson@geosyntec.com> wrote:
> > > Ben and Pierre,
> > >
> > > Thanks for the tips! Glad I chimed in here b/c I've definitely learned
> > something. I'm often pulling data out from a database, so null records
> > come back to me as None's. I'll be sure to set a CASE statement now that
> > fills the NULLs in with an obviously junk value that I can mask from now
> > on.
> > >
> > > -paul
> > Why not user your None as said junk value?
> > --Josh
> All my existing routines just throw values from the database (Postgres or
> MS SQL) cursor directly into numpy arrays. As Pierre pointed out, the None
> values force the array's dtype to object.
> In : x
> Out: array([2.5, 12.2, 5, None], dtype=object)
> I could easily mask the Nones, but switching it over to a value such as
> -99999 would let it me keep the dtype as float or integer. Probably not a
> big deal at the moment, but I'm trying to adopt as many best practices as
> possible so nothing comes back to bite me later on. (Negative values are
> pretty rare in environmental data, so I think it's a safe bet).
As a general rule, magic values can be very tricky to work with, and it is a
good idea to think out conventions. I was just bitten by a bug where a
previous developer wanted coordinate values of exactly 0.0 (yes, floating
point) to mean something special, but had no constraints for the domain.
So, once in a blue moon, a data point would get processed differently than
all the others, causing slightly odd analysis results.
So, my personal rule is to use NaNs whenever/where-ever possible because if
they sneak into a calculation unexpectedly, they will usually propagate such
that your program breaks -- which is a good thing. Bad results that appear
to be good is a bad thing. When you have to do integers, pick a magic value
that is impossible (e.g., negative value for temperature in Kelvins), and
document that magic value.
I hope this is helpful.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the SciPy-User