[NumPy-Tickets] [NumPy] #1616: numpy.loadtxt uses both file-like and iterator interfaces instead of just one

[NumPy Trac]

#1616: numpy.loadtxt uses both file-like and iterator interfaces instead of just
one
-------------------------+--------------------------------------------------
 Reporter:  zachrahan    |       Owner:  somebody
     Type:  enhancement  |      Status:  new     
 Priority:  normal       |   Milestone:  1.5.1   
Component:  Other        |     Version:  devel   
 Keywords:               |  
-------------------------+--------------------------------------------------
 numpy.loadtxt takes as input a filename or file-like object; for many
 useful tasks (like stopping reading lines after some specific delimiter)
 it would be easiest to just wrap a real file in some shim object that
 behaves as desired, rather than adding more options to loadtxt itself.

 However, loadtxt requires that file-like objects passed in expose both the
 iterator interface and a "readline" attribute. This makes the task
 unnecessarily hard, and makes it impossible to just pass in a generator.

 The following patch removes the use of "readline" without altering the
 behavior of the code at all.

 {{{
 Index: numpy/lib/npyio.py
 ===================================================================
 --- numpy/lib/npyio.py  (revision 8716)
 +++ numpy/lib/npyio.py  (working copy)
 @@ -597,10 +597,11 @@
              fh = bz2.BZ2File(fname)
          else:
              fh = open(fname, 'U')
 -    elif hasattr(fname, 'readline'):
 -        fh = fname
      else:
 -        raise ValueError('fname must be a string or file handle')
 +      try:
 +          fh = iter(fname)
 +      except:
 +          raise ValueError('fname must be a string or file handle')
      X = []

      def flatten_dtype(dt):
 @@ -633,14 +634,18 @@

          # Skip the first `skiprows` lines
          for i in xrange(skiprows):
 -            fh.readline()
 +            try:
 +                fh.next()
 +            except StopIteration:
 +                raise IOError('End-of-file reached before
 encountering data.')

          # Read until we find a line with some values, and use
          # it to estimate the number of columns, N.
          first_vals = None
          while not first_vals:
 -            first_line = fh.readline()
 -            if not first_line: # EOF reached
 +            try:
 +                first_line = fh.next()
 +            except StopIteration:
                  raise IOError('End-of-file reached before
 encountering data.')
              first_vals = split_line(first_line)
          N = len(usecols or first_vals)
 }}}

-- 
Ticket URL: <http://projects.scipy.org/numpy/ticket/1616>
NumPy <http://projects.scipy.org/numpy>
My example project