Tue Jul 20 18:17:42 CDT 2010
On Tue, Jul 20, 2010 at 3:12 PM, Benjamin Root <firstname.lastname@example.org> wrote:
> On Tue, Jul 20, 2010 at 12:24 PM, <PHobson@geosyntec.com> wrote:
>> > > Ben and Pierre,
>> > >
>> > > Thanks for the tips! Glad I chimed in here b/c I've definitely learned
>> > something. I'm often pulling data out from a database, so null records
>> > come back to me as None's. I'll be sure to set a CASE statement now that
>> > fills the NULLs in with an obviously junk value that I can mask from now
>> > on.
>> > >
>> > > -paul
>> > Why not user your None as said junk value?
>> > --Josh
>> All my existing routines just throw values from the database (Postgres or
>> MS SQL) cursor directly into numpy arrays. As Pierre pointed out, the None
>> values force the array's dtype to object.
>> In : x
>> Out: array([2.5, 12.2, 5, None], dtype=object)
>> I could easily mask the Nones, but switching it over to a value such as
>> -99999 would let it me keep the dtype as float or integer. Probably not a
>> big deal at the moment, but I'm trying to adopt as many best practices as
>> possible so nothing comes back to bite me later on. (Negative values are
>> pretty rare in environmental data, so I think it's a safe bet).
> As a general rule, magic values can be very tricky to work with, and it is a
> good idea to think out conventions. I was just bitten by a bug where a
> previous developer wanted coordinate values of exactly 0.0 (yes, floating
> point) to mean something special, but had no constraints for the domain.
> So, once in a blue moon, a data point would get processed differently than
> all the others, causing slightly odd analysis results.
> So, my personal rule is to use NaNs whenever/where-ever possible because if
> they sneak into a calculation unexpectedly, they will usually propagate such
> that your program breaks -- which is a good thing. Bad results that appear
> to be good is a bad thing. When you have to do integers, pick a magic value
> that is impossible (e.g., negative value for temperature in Kelvins), and
> document that magic value.
> I hope this is helpful.
> Ben Root
> SciPy-User mailing list
Hmm! Yeah, NaN would make more sense. >_<
More information about the SciPy-User