[Scipy-tickets] [SciPy] #1290: netcdf - memory error or other error during scipy.io.netcdf.NetCDFFile(...)

SciPy Trac scipy-tickets@scipy....
Tue Sep 28 09:14:16 CDT 2010


#1290: netcdf - memory error or other error during scipy.io.netcdf.NetCDFFile(...)
-----------------------+----------------------------------------------------
 Reporter:  hfrkessel  |       Owner:  somebody
     Type:  defect     |      Status:  new     
 Priority:  normal     |   Milestone:  0.9.0   
Component:  scipy.io   |     Version:  0.7.0   
 Keywords:             |  
-----------------------+----------------------------------------------------
 Symtom: scipy cannot read netcdf-files under Windows, an out of memory or
 other error is shown.

 Observation: netcdf-files are readable under Linux

 Reason: During the read of the netcdf-file the current read position gets
 corrupted when an mmap object is used.
 The implementation of the mmap object is different under Linux and
 Windows. The use of the Windows mmap object changes the file position in
 the refered file discriptor, while the Linux mmap object does not.

 Patch: The problem can be fixed in the class netcdf_file(object).
 The mmap object is used in the method _read_var_array() [line 442].

 In the 2 places where the mmap object is used, the current file position
 must saved and restored.

 ...
                 if self.use_mmap:
                     pos=self.fp.tell() # save position in file
                     mm = mmap(self.fp.fileno(), begin_+a_size,
 access=ACCESS_READ)
                     data = ndarray.__new__(ndarray, shape, dtype=dtype_,
                             buffer=mm, offset=begin_, order=0)
                     if self.fp.tell()!=pos:
                         print "Warning: ",self.fp.tell(),pos
                     self.fp.seek(pos) # restore position in file
 ...

 and

 ...
                 pos=self.fp.tell() # save position in file
                 mm = mmap(self.fp.fileno(),
 begin+self._recs*self._recsize, access=ACCESS_READ)
                 rec_array = ndarray.__new__(ndarray, (self._recs,),
 dtype=dtypes,
                         buffer=mm, offset=begin, order=0)
                 self.fp.seek(pos) # restore position in file
 ...

 This is a quick fix which will work under Windows and Linux. It might be
 useful to change the mmap object that it does not modifies the file
 pointer in the file the mmap object is based on.

-- 
Ticket URL: <http://projects.scipy.org/scipy/ticket/1290>
SciPy <http://www.scipy.org>
SciPy is open-source software for mathematics, science, and engineering.


More information about the Scipy-tickets mailing list