[NumPy-Tickets] [NumPy] #1660: ~2**32 byte tofile()/fromfile() limit in 64-bit Windows

[NumPy Trac]

#1660: ~2**32 byte tofile()/fromfile() limit in 64-bit Windows
-------------------------------------------------+--------------------------
 Reporter:  mspacek                              |       Owner:  somebody
     Type:  defect                               |      Status:  new     
 Priority:  normal                               |   Milestone:  2.0.0   
Component:  numpy.core                           |     Version:  1.5.0   
 Keywords:  tofile fromfile save 64-bit windows  |  
-------------------------------------------------+--------------------------
 I'm using Christoph Gohlke's amd64 builds
 (http://www.lfd.uci.edu/~gohlke/pythonlibs/#numpy) on 64-bit Windows 7,
 with the official Python 2.6.6 amd64 install. I can't tofile() or save()
 or fromfile() numpy arrays roughly > 2**32 bytes. For save() and tofile()
 Python hangs, using 100% CPU on one core. For fromfile(), it raises an
 IOError. This doesn't happen in 64-bit Linux on the same machine. load()
 seems to work on any size of file. Here are some examples, and some
 caveats:

 {{{
 # if .tofile() hangs, it hangs immediately:
 np.zeros(2**32+2**16, np.int8).tofile('test.npy') # hangs, file is 2**16
 bytes
 np.zeros(2**32+2**2,  np.int8).tofile('test.npy') # hangs, file is 0 bytes
 np.zeros(2**32+2,     np.int8).tofile('test.npy') # hangs, file is 0 bytes
 np.zeros(2**32+1,     np.int8).tofile('test.npy') # hangs, file is 0 bytes
 np.zeros(2**32,       np.int8).tofile('test.npy') # hangs, file is 0 bytes
 np.zeros(2**32-1,     np.int8).tofile('test.npy') # works

 # if np.save() hangs, it takes a while for it to hang:
 np.save('test.npy', np.zeros(2**33, np.int8)) # hangs, file is 2**32
 bytes, npy header is intact
 np.save('test.npy', np.zeros(2**33-1, np.int8)) # hangs, file is 2**32
 bytes, npy header is intact
 np.save('test.npy', np.zeros(2**32+2**31, np.int8)) # hangs, file is 2**31
 bytes, npy header is intact
 np.save('test.npy', np.zeros(2**32+2**16, np.int8)) # hangs, file is 2**16
 bytes, npy header is intact
 np.save('test.npy', np.zeros(2**32+2**12, np.int8)) # hangs, file is 2**12
 bytes, npy header is intact
 np.save('test.npy', np.zeros(2**32+2**11, np.int8)) # works
 np.save('test.npy', np.zeros(2**32+2**10, np.int8)) # works
 np.save('test.npy', np.zeros(2**32+2**8, np.int8)) # works
 np.save('test.npy', np.zeros(2**32+2**4, np.int8)) # works
 np.save('test.npy', np.zeros(2**32+2**2, np.int8)) # works
 np.save('test.npy', np.zeros(2**32+2, np.int8)) # works
 np.save('test.npy', np.zeros(2**32+1, np.int8)) # works
 np.save('test.npy', np.zeros(2**32, np.int8)) # works

 # I also generated these 3 large files successfully in 64-bit Linux...
 np.zeros(2**33, np.int8).tofile('test1.npy') # 8 GB
 np.zeros(2**32, np.int8).tofile('test2.npy') # 4 GB
 np.save('test3.npy', np.zeros(2**33, np.int8)) # 8 GB

 # ...and then tried reading them back in 64-bit Win7:
 a = np.fromfile('test1.npy', dtype=np.int8) # IOError: could not seek in
 file
 a = np.fromfile('test2.npy', dtype=np.int8) # IOError: could not seek in
 file
 a = np.load('test3.npy') # strangely enough, this works fine, even though
 it's 8 GB!
 }}}

 I have this problem regardless of which of Christoph's amd64 numpy builds
 (1.4.1, 1.5.0, 1.5.1RC1, MKL or non-MKL) I use. This happens on both my
 64-bit Win7 i7 12 GB machine, and on another machine running 64-bit WinXP.
 Christoph has also confirmed this on Python 2.5 through 3.1 and says there
 is nothing special about his builds, and that this will affect all 64-bit
 Windows distributions of numpy (EPD, ActiveState, etc).

-- 
Ticket URL: <http://projects.scipy.org/numpy/ticket/1660>
NumPy <http://projects.scipy.org/numpy>
My example project