[Numpy-discussion] Efficient way to load a 1Gb file?

Russell E. Owen rowen@uw....
Mon Aug 15 15:25:50 CDT 2011


In article 
<CA+RwOBWjyY_abjijnxEPkSeRaeom608uiMYwffGaG-6XDgSdPw@mail.gmail.com>,
 Torgil Svensson <torgil.svensson@gmail.com> wrote:

> Try the fromiter function, that will allow you to pass an iterator
> which can read the file line by line and not preload the whole file.
> 
> file_iterator = iter(open('filename.txt')
> line_parser = lambda x: map(float,x.split('\t'))
> a=np.fromiter(itertools.imap(line_parser,file_iterator),dtype=float)
> 
> You have also the option to iterate the file twice and pass the
> "count" argument.

Thanks. That sounds great!

-- RUssell



More information about the NumPy-Discussion mailing list