[Numpy-discussion] seeking advice on a fast string->array conversion

Darren Dale dsdale24@gmail....
Fri Nov 19 10:29:19 CST 2010


On Tue, Nov 16, 2010 at 10:31 AM, Darren Dale <dsdale24@gmail.com> wrote:
> On Tue, Nov 16, 2010 at 9:55 AM, Pauli Virtanen <pav@iki.fi> wrote:
>> Tue, 16 Nov 2010 09:41:04 -0500, Darren Dale wrote:
>> [clip]
>>> That loop takes 0.33 seconds to execute, which is a good start. I need
>>> some help converting this example to return an actual numpy array. Could
>>> anyone please offer a suggestion?
>>
>> Easiest way is probably to use ndarray buffers and resize them when
>> needed. For example:
>>
>> https://github.com/pv/scipy-work/blob/enh/interpnd-smooth/scipy/spatial/qhull.pyx#L980
>
> Thank you Pauli. That makes it *incredibly* simple:
>
> import time
> cimport numpy as np
> import numpy as np
>
> cdef extern from 'stdlib.h':
>    double atof(char*)
>
>
> def test():
>    py_string = '100'
>    cdef char* c_string = py_string
>    cdef int i, j
>    cdef double val
>    i = 0
>    j = 2048*1200
>    cdef np.ndarray[np.float64_t, ndim=1] ret
>
>    ret_arr = np.empty((2048*1200,), dtype=np.float64)
>    ret = ret_arr
>
>    d = time.time()
>    while i<j:
>        c_string = py_string
>        ret[i] = atof(c_string)
>        i += 1
>    ret_arr.shape = (1200, 2048)
>    print ret_arr, ret_arr.shape, time.time()-d
>
> The loop now takes only 0.11 seconds to execute. Thanks again.
>

One follow-up issue: I can't cythonize this code for python-3. I've
installed numpy with the most recent changes to the 1.5.x maintenance
branch, then re-installed cython-0.13, but when I run "python3
setup.py build_ext --inplace" with this setup script:

from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext

import numpy

setup(
    cmdclass = {'build_ext': build_ext},
    ext_modules = [
        Extension(
            "test_open", ["test_open.pyx"], include_dirs=[numpy.get_include()]
            )
        ]
)


I get the following error. Any suggestions what I need to fix, or
should I report it to the cython list?

$ python3 setup.py build_ext --inplace
running build_ext
cythoning test_open.pyx to test_open.c

Error converting Pyrex file to C:
------------------------------------------------------------
...
        # For use in situations where ndarray can't replace PyArrayObject*,
        # like PyArrayObject**.
        pass

    ctypedef class numpy.ndarray [object PyArrayObject]:
        cdef __cythonbufferdefaults__ = {"mode": "strided"}
                                                ^
------------------------------------------------------------

/Users/darren/.local/lib/python3.1/site-packages/Cython/Includes/numpy.pxd:173:49:
"mode" is not a buffer option

Error converting Pyrex file to C:
------------------------------------------------------------
...
   cdef char* c_string = py_string
   cdef int i, j
   cdef double val
   i = 0
   j = 2048*1200
   cdef np.ndarray[np.float64_t, ndim=1] ret
       ^
------------------------------------------------------------

/Users/darren/temp/test/test_open.pyx:16:8: 'ndarray' is not a type identifier
building 'test_open' extension
/usr/bin/gcc-4.2 -fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g
-fwrapv -O3 -Wall -Wstrict-prototypes
-I/Users/darren/.local/lib/python3.1/site-packages/numpy/core/include
-I/opt/local/Library/Frameworks/Python.framework/Versions/3.1/include/python3.1
-c test_open.c -o build/temp.macosx-10.6-x86_64-3.1/test_open.o
test_open.c:1:2: error: #error Do not use this file, it is the result
of a failed Cython compilation.
error: command '/usr/bin/gcc-4.2' failed with exit status 1


More information about the NumPy-Discussion mailing list