[Numpy-discussion] reading large files

Mike Ressler mike.ressler at alum.mit.edu
Fri Sep 22 14:37:37 CDT 2006

On 9/22/06, Bill Baxter <wbaxter at gmail.com> wrote:
> Do you also have a 64-bit processor?  Just checking since you didn't
> mention it.
> --bb
> On 9/22/06, Svein-Erik.Hamran at ffi.no <Svein-Erik.Hamran at ffi.no> wrote:
> > I would like to read files >2Gbyte. From earlier posting I believed it
> > should be possible with python 2.5.
> >
> > I am getting MemoryError when trying to read a file larger than 2Gb.
> >
> > I am using python2.5 and numpy1.0rc1.

Also, how much memory do you have? You will have to use memory mapping as
well if the file size is greater than your memory size.

But you should be hopeful. I have been working with > 10 GB files for some
time using beta versions of Python 2.5 and numpy-1.0 on a 64-bit AMD machine
with 4 GB of memory.


mike.ressler at alum.mit.edu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://projects.scipy.org/pipermail/numpy-discussion/attachments/20060922/aa6590da/attachment-0001.html 

More information about the Numpy-discussion mailing list