[AstroPy] patched pyfits 2.2.2

Erin Sheldon erin.sheldon@gmail....
Tue Oct 20 15:27:44 CDT 2009


Hi All -

I've been maintaining a patched version of pyfits with some extended
functionality for dealing with structured arrays (aka recarrays, arrays with
fields).  These patches are primarily related to the writing and reading of
binary tables in a straightforward way using writeto/getdata.  In my work I
regularly create massive amounts of data and write it to FITS binary tables, so
this is of particular importance to me.  Here is a simple example of the
new functionality.

e.g

    # write an array with field directly to a fits file
    st = numpy.zeros(3, [('f1','f4'),('f2','S10'),('f3','2f8')])

    st['f1'] = [1,3,5]
    st['f2'] = ['hello','world','byebye']
    st['f3'] = numpy.random.random(st['f3'].shape)

    pyfits.writeto('testnew.fits', st)

    # read FITS data and write it back out to a different file
    data = pyfits.getdata('testnew.fits', ext=2, lower=True)
    pyfits.writeto('tmpfile.fits', data)

A full description of the changes is at the bottom of this message, and labeled
in the code with E.S.S.

I've submitted patches to the pyfits maintainers but they are quite busy and
won't be able to test or incorporate these patches any time soon.

So I'm asking for some help testing these.  My colleagues and I have been using
these patches for a year or so but we need more eyes.  The patches pass all the
unit tests under /test.  I've also added my own test testESS.py there.  If you
are interested, please download this tar ball and help us test.

http://www.cosmo.bnl.gov/www/esheldon/code/python/pyfits-2.2.2espatch-v1.tar.gz

Erin Sheldon
Brookhaven National Laboratory

Description of changes:

NP_pyfits.pro:
Patched FITS_rec class to allow views from ordinary numpy arrays/recarray

writeto: Can now write arrays with fields (recarray, structured arrays)
    directly to a file.  Calls _array_with_fields_writeto when appropriate.
    Also added check for an append= keyword for appending.
append: Call _array_with_fields_writeto when appropriate.
getdata: lower= and upper= keywords to change case of field names.

_extract_format:  A new method to get types.  Deals properly with
    multi-dimensional arrays.  This still needs to use TDIM to reshape
    into a NxM but at least the data is now characterized.
    TODO: implement shape info from TDIM
_convert_format:  Fixed error dealing with multi-dimensional arrays.
    TODO: implement shape info from TDIM

_array_with_fields_writeto:  New function to deal with writing an input
    ndarray,recarray, or fits.rec.recarray directly to a file, possibly
    appending.
    TODO: Deal with updating a header

_stat_filename_or_fileobj(filename): utility function to get info about the
    input file name or object.

rec.py:
fromfile():  Use a view instead of recarray() which was giving errors
    for large arrays.  Don't fully understand this.


Examples:


    st = numpy.zeros(3, [('f1','f4'),('f2','S10'),('f3','2f8')])

    numpy.random.seed(35)
    st['f1'] = [1,3,5]
    st['f2'] = ['hello','world','byebye']
    st['f3'] = numpy.random.random(st['f3'].shape)
    print(st)

    pyfits.writeto('testnew.fits', st)


    st['f1'] = [8,12,55]
    st['f2'] = ['stuff','things','blah']
    st['f3'] = numpy.random.random(st['f3'].shape)

    print(st)

    pyfits.writeto('testnew.fits', st, append=True)

    data = pyfits.getdata('testnew.fits', ext=2, lower=True)
    print(data)

    pyfits.writeto('tmpfile.fits', data)


More information about the AstroPy mailing list