[SciPy-user] (off topic) experience with pytables and masked arrays?
mattknox_ca at hotmail.com
Mon Jan 22 12:21:00 CST 2007
Sorry for the off topic post... but this seemed like the most likely spot to find a response...
I spoke with Francesc about this, and Pytables does not support masked arrays (and won't be in the near future). It's not too hard to envision some possibilities about how one would create work arounds for doing this kind of thing... but I thought I would just ask to see if anyone has already tried this on their own.
So... does anyone that works with masked arrays use pytables for storage? And if so, have any code they would be willing to share?
My initial thoughts on how one would do this would be to pick a "special" value for each data type (eg. sys.maxint for int32, NaN for float, etc...), and replace masked values with this value prior to writing. And conversely, mask these values when reading (and probably not only mask them, but replace them in the underlying data array with safer values for performing operations (ie. array.fill_value)). Several additional attributes would have to be stored along with the array: fill_value , some kind of flag to indicate it is a masked array, etc...
In a perfect world this would be done within the pytables core, and then all the array related functions and methods in pytables would just work with masked arrays directly, requiring no special handling from the user, but that isn't going to happen anytime soon. So if anyone has built their own functions for dealing with this situation (or has some thoughts on strategies for dealing with missing data points in pytables in general), I would love to hear about it.
- Matt Knox
More information about the SciPy-user