[NumPy-Tickets] [NumPy] #1428: genfromtxt problem with long integers

[NumPy Trac]

#1428: genfromtxt problem with long integers
-----------------------+----------------------------------------------------
 Reporter:  josefpktd  |       Owner:  somebody
     Type:  defect     |      Status:  new     
 Priority:  normal     |   Milestone:          
Component:  Other      |     Version:          
 Keywords:             |  
-----------------------+----------------------------------------------------
 It seems genfromtxt has problems identifying long integers (at
 least on Windows 32)

 I have a csv file with large integers:

 {{{
 >>> np.array(4160680000,int)
 Traceback (most recent call last):
  File "<pyshell#4>", line 1, in <module>
    np.array(4160680000,int)
 OverflowError: long int too large to convert to int
 }}}

 genfromtxt fails with non-informative exception

 {{{
 >>> s = '''Date,Open,High,Low,Close,Volume,Adj Close
 ... 2010-02-12,1075.95,1077.81,1062.97,1075.51,4160680000,1075.51
 ... 2010-02-11,1067.10,1080.04,1060.59,1078.47,4400870000,1078.47'''


 >>> sh = StringIO(s)
 >>> data = np.genfromtxt(sh, delimiter=",", dtype=None, names=True)
 Traceback (most recent call last):
  File "C:\Programs\Python25\Lib\site-packages\numpy\lib\io.py", line
 1367, in genfromtxt
    output = np.array(data, dtype=ddtype)
 TypeError: expected a readable buffer object
 }}}

 same with explicit dtypes using int for the long integer

 {{{
 >>> dt=
 [('','S10'),('',float),('',float),('',float),('',float),('',int),('',float)]
 >>> sh = StringIO(s)
 >>> data = np.genfromtxt(sh, delimiter=",", dtype=dt, names=True)
 Traceback (most recent call last):
  File "C:\Programs\Python25\Lib\site-packages\numpy\lib\io.py", line
 1388, in genfromtxt
    rows = np.array(data, dtype=[('', _) for _ in dtype_flat])
 TypeError: expected a readable buffer object
 }}}


 using float works:

 {{{
 >>> dt=
 [('','S10'),('',float),('',float),('',float),('',float),('',float),('',float)]
 >>> sh = StringIO(s)
 >>> data = np.genfromtxt(sh, delimiter=",", dtype=dt, names=True)
 >>> data
 array([ ('2010-02-12', 1075.95, 1077.8099999999999, 1062.97, 1075.51,
 4160680000.0, 1075.51),
       ('2010-02-11', 1067.0999999999999, 1080.04, 1060.5899999999999,
 1078.47, 4400870000.0, 1078.47)],
      dtype=[('Date', '|S10'), ('Open', '<f8'), ('High', '<f8'),
 ('Low', '<f8'), ('Close', '<f8'), ('Volume', '<f8'), ('Adj_Close',
 '<f8')])
 >>>
 }}}

-- 
Ticket URL: <http://projects.scipy.org/numpy/ticket/1428>
NumPy <http://projects.scipy.org/numpy>
My example project