[NumPy-Tickets] [NumPy] #2247: genfromtxt/recfromcsv problem with long integers

[NumPy Trac]

#2247: genfromtxt/recfromcsv problem with long integers
------------------------+---------------------------------------------------
 Reporter:  b0rek       |       Owner:  somebody   
     Type:  defect      |      Status:  new        
 Priority:  normal      |   Milestone:  Unscheduled
Component:  numpy.core  |     Version:  1.6.1      
 Keywords:              |  
------------------------+---------------------------------------------------
 Ticket #1428 is no solution, when you can not predict values in opened
 file.

 {{{
 >>> sss = '''Date,Open,High,Low,Close,Volume,Adj
 ... 2010-02-12,1075.95,1077.81,1062.97,1075.51,4160680000,1075.51
 ... 2010-02-11,1067.10,1080.04,1060.59,1078.47,4400870000,1078.47'''
 >>> import StringIO
 >>> sh = StringIO.StringIO(s)
 >>> data = np.genfromtxt(sh, delimiter=",", dtype=None, names=True)
 Traceback (most recent call last):
   File "<interactive input>", line 1, in <module>
   File "D:\PP273\App\lib\site-packages\numpy\lib\npyio.py", line 1613, in
 genfromtxt
     output = np.array(data, dtype=ddtype)
 OverflowError: Python int too large to convert to C long
 }}}

 There should be no OverflowError for correct values.

 Temporary solution is to change in class StringConverter._mapper
 (numpy/lib/_iotools.py) from

 {{{
     _mapper = [(nx.bool_, str2bool, False),
                (nx.integer, int, -1),
                (nx.floating, float, nx.nan),
                (complex, _bytes_to_complex, nx.nan + 0j),
                (nx.string_, bytes, asbytes('???'))]
 }}}
 to
 {{{
     _mapper = [(nx.bool_, str2bool, False),
                (nx.int64, int, -1),
                (nx.floating, float, nx.nan),
                (complex, _bytes_to_complex, nx.nan + 0j),
                (nx.string_, bytes, asbytes('???'))]
 }}}

 but it is not optimal. Converter should detect int8, int16 etc.

-- 
Ticket URL: <http://projects.scipy.org/numpy/ticket/2247>
NumPy <http://projects.scipy.org/numpy>
My example project