[NumPy-Tickets] [NumPy] #1660: ~2**32 byte tofile()/fromfile() limit in 64-bit Windows

[NumPy Trac]

#1660: ~2**32 byte tofile()/fromfile() limit in 64-bit Windows
-------------------------------------------------+--------------------------
 Reporter:  mspacek                              |       Owner:  somebody
     Type:  defect                               |      Status:  new     
 Priority:  normal                               |   Milestone:  2.0.0   
Component:  numpy.core                           |     Version:  1.5.0   
 Keywords:  tofile fromfile save 64-bit windows  |  
-------------------------------------------------+--------------------------

Comment(by mspacek):

 Well, that's depressing. I should just switch to linux already. It looks
 like fseek and ftell are 32 bit in msvc, even in Win64. Perhaps _fseeki64
 and _ftelli64 need to be used instead?:

 http://msdn.microsoft.com/en-us/library/75yw9bf3%28v=VS.90%29.aspx

 http://msdn.microsoft.com/en-us/library/0ys3hc0b%28v=VS.90%29.aspx

 http://www.firstobject.com/fseeki64-ftelli64-in-vc++.htm

 This might explain why "IOError: could not seek in file" is thrown in
 /numpy/core/src/multiarray/ctors.c line 3037 when I call fromfile():


 {{{
 static PyArrayObject *
 array_fromfile_binary(FILE *fp, PyArray_Descr *dtype, intp num, size_t
 *nread)
 {
     PyArrayObject *r;
     intp start, numbytes;

     if (num < 0) {
         int fail = 0;

         start = (intp )ftell(fp);
         if (start < 0) {
             fail = 1;
         }
         if (fseek(fp, 0, SEEK_END) < 0) {
             fail = 1;
         }
         numbytes = (intp) ftell(fp);
         if (numbytes < 0) {
             fail = 1;
         }
         numbytes -= start;
         if (fseek(fp, start, SEEK_SET) < 0) {
             fail = 1;
         }
         if (fail) {
             PyErr_SetString(PyExc_IOError,
                             "could not seek in file");
             Py_DECREF(dtype);
             return NULL;
         }
         num = numbytes / dtype->elsize;
     }
     r = (PyArrayObject *)PyArray_NewFromDescr(&PyArray_Type,
                                               dtype,
                                               1, &num,
                                               NULL, NULL,
                                               0, NULL);
     if (r == NULL) {
         return NULL;
     }
     NPY_BEGIN_ALLOW_THREADS;
     *nread = fread(r->data, dtype->elsize, num, fp);
     NPY_END_ALLOW_THREADS;
     return r;
 }
 }}}

 However, I don't understand why the error comes up when I call fromfile()
 directly, but not when I call it indirectly via np.load()

-- 
Ticket URL: <http://projects.scipy.org/numpy/ticket/1660#comment:4>
NumPy <http://projects.scipy.org/numpy>
My example project