[AstroPy] reading one line from many small fits files
Tue Jul 31 11:13:40 CDT 2012
On 31.07.2012, at 5:52PM, Erik Bray <firstname.lastname@example.org> wrote:
> It probably shouldn't matter whether or not he's calling close(), but
> the question about BSCALE/BZERO is possibly relevant. Is the data
> you're reading from an image or a table? If it's an image, then as
> Derek wrote PyFITS is still pretty inefficient, in that it will
> transform the entire image, even if using mmap (which is the default now
> by the way). I have plans for overhauling this but it hasn't been a high
> priority for the most part. You can also turn off image scaling if you
> use do_not_scale_image_data=True when opening the file. That might
> speed things up.
> This is one area where using the .section feature on Image HDUs might
> still be useful. For example:
> result[i] = photo.section[x[otherfield] - 1]
> In PyFITS 3.1 that has improved support for scaling just sections of the
> file, which didn't work well before. That might also be faster.
> Of course this is all a moot point if this is not a scaled image. In
> any case, opening a file and reading a single row out of the data should
> not generally take as long as 1 second--especially if they're small files.
indeed the question was about tables, I just still had the discussion about automatic
scaling and the potential type conversion coming with it in mind. And the docs mention
`FITS_rec` is the data part of a table HDU's data part. This is a
layer over the `recarray`, so we can deal with scaled columns.
so I was wondering if some automatic scaling features might have been added
(or be in testing) in 3.1 svn.
More information about the AstroPy