[SciPy-User] genfromtxt with missing fields

Ian Stokes-Rees ijstokes@hkl.hms.harvard....
Fri Jun 18 15:49:14 CDT 2010


> Since you can specify which columns to skip, then there is a newline
> character only if it's the last (few) column(s) that's missing
> correct?  If not the last column then it's just \t\t?  My suggestion
> would be to iterate through the ASCII file, split each line on the
> delimiter, and add tabs for these last missing entries if needed, if I
> understand you correctly.
>   

Yes, I can re-process the data inside my script, but I was hoping there
was some clever numpy (or scipy.io) oriented way to deal with this, and
there is...

I should have read the exception better and looked at the full docs for
genfromtxt.  The answer is right there: the boolean flag "invalid_raise"
will cause malformed lines to be skipped.  The exception even shows
where this will be applied.

Ian

numpy/lib/io.py in genfromtxt(fname, dtype, comments, delimiter,
skiprows, skip_header, skip_footer, converters, missing, missing_values,
filling_values, usecols, names, excludelist, deletechars, autostrip,
case_sensitive, defaultfmt, unpack, usemask, loose, invalid_raise)
   1319             # Raise an exception ?

   1320             if invalid_raise:
-> 1321                 raise ValueError(errmsg)
   1322             # Issue a warning ?

   1323             else:


More information about the SciPy-User mailing list