[SciPy-User] FortranFile and large records

Vincent Favre-Nicolin vincefn@users.sourceforge....
Thu Jan 19 10:16:50 CST 2012


        Hi,

    This week I had to import large data from a Fortran 'record' file.
For this I naturally used FortranFile
(http://www.scipy.org/Cookbook/FortranIO/FortranFile).

    Unfortunately, it turned out that when a record is larger than
2,147,483,639 bytes (a bit short of 2^31), the 'record' where the data
is stored is split into sub-records.
    This took me some time to figure out, since I don't use Fortran that
much, and the only symptom was a negative size for the record... Which I
originally interpreted as a type (signed/unsigned) or byteswapping issue....

    Anyway, once I got what was going wrong (from Intel's
documentation),  I modified the "readRecord" function in FortranFile to
be able to read a complete record when such a case is encountered. The
corresponding code is below (not very efficient memorywise but working
as long as you have >=24Gb of memory, to perform directly a readReals()).

    I'm not sure if there is another way to do this (in numpy),
hopefully it will be useful for someone else. Neil if you think it is
OK, maybe the wiki page can be updated. And thanks again for FortranFile
which is really useful !


            Vincent


    def readRecord(self):
        """Read a single fortran record"""
        l = self._read_check()
        data_str=''
        while l<0:
          # Handle sub-records, for lengths>2,147,483,639 bytes
          # cf: Intel Fortran Compiler User and Reference Guides, page
217-218
          # (doc number 304970-006US)
          data_str+=self._read_exactly(abs(l))
          check_size = self._read_check()
          if abs(check_size) != abs(l):
              raise IOError('Error reading record from data file')
          l = self._read_check()
        data_str += self._read_exactly(abs(l))
        check_size = self._read_check()
        if abs(check_size) != abs(l):
            raise IOError('Error reading record from data file')
        return data_str

-- 
Vincent Favre-Nicolin                    http://inac.cea.fr

CEA/Grenoble              Institut Nanosciences & Cryogénie
Laboratoire SP2M/Nano-structures et Rayonnement Synchrotron
17, rue des Martyrs
38054 Grenoble Cedex 9 - France

Université Joseph Fourier        http://www.ujf-grenoble.fr

tél: (+33) 4 38 78 95 40           fax: (+33) 4 38 78 51 38



More information about the SciPy-User mailing list