[SciPy-user] Scipy weave errors: undefined symbols when importing compiled extension

Craig Finch oanjao@yahoo....
Wed Jul 8 13:51:36 CDT 2009


Progress!  I asked on the Intel forums, and they pointed out that I can unmangle the symbol name using "c++filt _ZNSt8ios_base4InitC1Ev".  This corresponds to "std::ios_base::Init::Init()" which is in the C++ standard library.  It appears that the code was not being linked with C++ standard library, and I was able to prevent this error by adding "-lstdc++" to the linker command.  At this point, I working copies of Python, Numpy, and Scipy built entirely with Intel tools.


However, I had to "hard code" this flag into the linker command in the file numpy/distutils/intelccompiler.py.  It just didn't work when I added it to the environment variable LDFLAGS (and yes, I double-checked the environment variable to make sure it was set correctly).  Because of this, I'd say I found a hackish workaround more than a solution.  Any suggestions about the "right" way to do this?

Second problem: Scipy now crashes when I try to run the unit tests.  Below is the partial output when I attempt to run the unit tests.  I'm also going to ask about this on the Intel MKL forums, but I'd appreciate any suggestions from you about how to go about diagnosing this problem.

   Craig

Python 2.5.4 (r254:67916, Jul  6 2009, 19:27:14)
[GCC Intel(R) C++ gcc 4.1 mode] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import scipy; scipy.test()
Running unit tests for scipy
NumPy version 1.3.0
NumPy is installed in /home/cfinch/lib/python2.5/site-packages/numpy
SciPy version 0.7.0
SciPy is installed in /home/cfinch/lib/python2.5/site-packages/scipy
Python version 2.5.4 (r254:67916, Jul  6 2009, 19:27:14) [GCC Intel(R) C++ gcc 4.1 mode]
nose version 0.11.1
...
...........................*** glibc detected *** python: free(): invalid next s
ize (fast): 0x0000000010781a50 ***
======= Backtrace: =========
/lib64/libc.so.6[0x315326e8a0]
/lib64/libc.so.6(cfree+0x8c)[0x3153271fbc]
/home/cfinch/lib/python2.5/site-packages/numpy/core/multiarray.so[0x2aaaaebc6827
] 
/home/cfinch/lib/python2.5/site-packages/scipy/linalg/flapack.so[0x2aaab702b927]
/home/cfinch/lib/python2.5/site-packages/scipy/linalg/flapack.so[0x2aaab703937b]
python(PyObject_Call+0x15)[0x417c65]
python[0x4c9c07]
python(PyEval_EvalFrameEx+0xe1e)[0x4c28a0]
python[0x4ca369]
python(PyEval_EvalFrameEx+0xe1e)[0x4c28a0]
python(PyEval_EvalCodeEx+0x5a2)[0x4cb7ee]
python[0x532ef5]
python(PyObject_Call+0x15)[0x417c65]
python(PyEval_EvalFrameEx+0x8ba)[0x4c233c]
python[0x4caa0f]
python(PyEval_EvalFrameEx+0xe1e)[0x4c28a0]
python(PyEval_EvalCodeEx+0x5a2)[0x4cb7ee]
python[0x532ef5]
python(PyObject_Call+0x15)[0x417c65]
python(PyEval_EvalFrameEx+0x8ba)[0x4c233c]
python(PyEval_EvalCodeEx+0x5a2)[0x4cb7ee]
python[0x532ef5]
python(PyObject_Call+0x15)[0x417c65]
python[0x421bcd]
python(PyObject_Call+0x15)[0x417c65]
python[0x48c28f]
python(PyObject_Call+0x15)[0x417c65]
python[0x4c9c07]
python(PyEval_EvalFrameEx+0xe1e)[0x4c28a0]
python[0x4caa0f]
python(PyEval_EvalFrameEx+0xe1e)[0x4c28a0]
python(PyEval_EvalCodeEx+0x5a2)[0x4cb7ee]
python[0x532ef5]
python(PyObject_Call+0x15)[0x417c65]
python(PyEval_EvalFrameEx+0x8ba)[0x4c233c]
python(PyEval_EvalCodeEx+0x5a2)[0x4cb7ee]
python[0x532ef5]
python(PyObject_Call+0x15)[0x417c65]
python[0x421bcd]
python(PyObject_Call+0x15)[0x417c65]
python[0x48c28f]
python(PyObject_Call+0x15)[0x417c65]
python[0x4c9c07]
python(PyEval_EvalFrameEx+0xe1e)[0x4c28a0]
python(PyEval_EvalCodeEx+0x5a2)[0x4cb7ee]
python[0x532ef5]
python(PyObject_Call+0x15)[0x417c65]
python(PyEval_EvalFrameEx+0x8ba)[0x4c233c]
python(PyEval_EvalCodeEx+0x5a2)[0x4cb7ee]
python[0x532ef5]
python(PyObject_Call+0x15)[0x417c65]
python[0x421bcd]
python(PyObject_Call+0x15)[0x417c65]
python[0x48c28f]
python(PyObject_Call+0x15)[0x417c65]
python[0x4c9c07]
python(PyEval_EvalFrameEx+0xe1e)[0x4c28a0]
python(PyEval_EvalCodeEx+0x5a2)[0x4cb7ee]
python[0x532ef5]
python(PyObject_Call+0x15)[0x417c65]
python(PyEval_EvalFrameEx+0x8ba)[0x4c233c]
python(PyEval_EvalCodeEx+0x5a2)[0x4cb7ee]
python[0x532ef5]

======= Memory map: ========
00400000-005e2000 r-xp 00000000 00:12 97412                              /gpfs/fs0/work/cfinch/bin/python
007e2000-0081a000 rwxp 001e2000 00:12 97412                              /gpfs/fs0/work/cfinch/bin/python
0081a000-00822000 rwxp 0081a000 00:00 0
0ede6000-10c04000 rwxp 0ede6000 00:00 0
40000000-40001000 ---p 40000000 00:00 0
40001000-40011000 rwxp 40001000 00:00 0
40011000-40012000 ---p 40011000 00:00 0
40012000-4040a000 rwxp 40012000 00:00 0
4040a000-4040b000 ---p 4040a000 00:00 0
4040b000-40803000 rwxp 4040b000 00:00 0
40803000-40804000 ---p 40803000 00:00 0
40804000-40bfc000 rwxp 40804000 00:00 0
40bfc000-40bfd000 ---p 40bfc000 00:00 0
40bfd000-40ff6000 rwxp 40bfd000 00:00 0
40ff6000-40ff7000 ---p 40ff6000 00:00 0
40ff7000-413f0000 rwxp 40ff7000 00:00 0
413f0000-413f1000 ---p 413f0000 00:00 0
413f1000-417ea000 rwxp 413f1000 00:00 0
417ea000-417eb000 ---p 417ea000 00:00 0
417eb000-41be4000 rwxp 417eb000 00:00 0
3152200000-315221a000 r-xp 00000000 08:03 131368                         /lib64/ld-2.5.so
3152419000-315241a000 r-xp 00019000 08:03 131368                         /lib64/ld-2.5.so
315241a000-315241b000 rwxp 0001a000 08:03 131368                         /lib64/ld-2.5.so
3152600000-3152614000 r-xp 00000000 08:03 24085282                       /usr/lib64/libz.so.1.2
.3
3152614000-3152813000 ---p 00014000 08:03 24085282                       /usr/lib64/libz.so.1.2
.3
3152813000-3152814000 rwxp 00013000 08:03 24085282                       /usr/lib64/libz.so.1.2
.3
3152a00000-3152a02000 r-xp 00000000 08:03 131373                         /lib64/libdl-2.5.so
3152a02000-3152c02000 ---p 00002000 08:03 131373                         /lib64/libdl-2.5.so
3152c02000-3152c03000 r-xp 00002000 08:03 131373                         /lib64/libdl-2.5.so
3152c03000-3152c04000 rwxp 00003000 08:03 131373                         /lib64/libdl-2.5.so
3152e00000-3152e35000 r-xp 00000000 08:03 24084785                       /usr/lib64/libreadline
.so.5.1
3152e35000-3153034000 ---p 00035000 08:03 24084785                       /usr/lib64/libreadline
.so.5.1
3153034000-315303c000 rwxp 00034000 08:03 24084785                       /usr/lib64/libreadline
.so.5.1
315303c000-315303d000 rwxp 315303c000 00:00 0
3153200000-3153344000 r-xp 00000000 08:03 131369                         /lib64/libc-2.5.so
3153344000-3153544000 ---p 00144000 08:03 131369                         /lib64/libc-2.5.so
3153544000-3153548000 r-xp 00144000 08:03 131369                         /lib64/libc-2.5.so

3153548000-3153549000 rwxp 00148000 08:03 131369                         /lib64/libc-2.5.so
3153549000-315354e000 rwxp 3153549000 00:00 0
3153600000-3153682000 r-xp 00000000 08:03 131374                         /lib64/libm-2.5.so
3153682000-3153881000 ---p 00082000 08:03 131374                         /lib64/libm-2.5.so
3153881000-3153882000 r-xp 00081000 08:03 131374                         /lib64/libm-2.5.so
3153882000-3153883000 rwxp 00082000 08:03 131374                         /lib64/libm-2.5.so
3153a00000-3153a15000 r-xp 00000000 08:03 131372                         /lib64/libpthread-2.5.so
3153a15000-3153c14000 ---p 00015000 08:03 131372                         /lib64/libpthread-2.5.so
3153c14000-3153c15000 r-xp 00014000 08:03 131372                         /lib64/libpthread-2.5.so
3153c15000-3153c16000 rwxp 00015000 08:03 131372                         /lib64/libpthread-2.5.so
3153c16000-3153c1a000 rwxp 3153c16000 00:00 0
3156e00000-3156e0d000 r-xp 00000000 08:03 131375                         /lib64/libgcc_s-4.1.1-20070105.so.1
3156e0d000-315700c000 ---p 0000d000 08:03 131375                         /lib64/libgcc_s-4.1.1-20070105.so.1
315700c000-315700d000 rwxp 0000c000 08:03 131375                         /lib64/libgcc_s-4.1.1-20070105.so.1
3157200000-31572e6000 r-xp 00000000 08:03 24085283                       /usr/lib64/libstdc++.so.6.0.8
31572e6000-31574e5000 ---p 000e6000 08:03 24085283                       /usr/lib64/libstdc++.so.6.0.8
31574e5000-31574eb000 r-xp 000e5000 08:03 24085283                       /usr/lib64/libstdc++.so.6.0.8
31574eb000-31574ee000 rwxp 000eb000 08:03 24085283                       /usr/lib64/libstdc++.so.6.0.8
31574ee000-3157500000 rwxp 31574ee000 00:00 0
315c000000-315c125000 r-xp 00000000 08:03 131380                         /lib64/libcrypto.so.0.9.8b
315c125000-315c324000 ---p 00125000 08:03 131380                         /lib64/libcrypto.so.0.9.8b
315c324000-315c343000 rwxp 00124000 08:03 131380                         /lib64/libcrypto.so.0.9.8b
315c343000-315c347000 rwxp 315c343000 00:00 0
315c400000-315c411000 r-xp 00000000 08:03 131378                         /lib64/libresolv-2.5.so
315c411000-315c611000 ---p 00011000 08:03 131378                         /lib64/libresolv-2.5.so
315c611000-315c612000 r-xp 00011000 08:03 131378                         /lib64/libresolv-2.5.so
315c612000-315c613000 rwxp 00012000 08:03 131378                         /lib64/libresolv-2.5.so
315c613000-315c615000 rwxp 315c613000 00:00 0
315d000000-315d002000 r-xp 00000000 08:03 Aborted



----- Original Message ----
From: Craig Finch <oanjao@yahoo.com>
To: David Cournapeau <cournape@gmail.com>; SciPy Users List <scipy-user@scipy.org>
Sent: Monday, July 6, 2009 7:16:42 PM
Subject: Re: [SciPy-user] Scipy weave errors: undefined symbols when importing  compiled extension

I've been dealing with issues like this, on this one system, for a year now.  At one point, I had a working combination of python, numpy, and scipy.  I tried so many things that I can't really remember how I got to that point.  I think I used gcc to build scipy 0.6 because I couldn't get it to work with icc, and I linked it with the Intel MKL successfully.  However, when I upgraded to scipy-0.7.0 I am again facing this problem.

The latest thing I tried was to build both Numpy and Scipy with the Intel C and Fortran compilers (icc and ifort).  Numpy builds fine and passes all its unit tests.  Scipy builds without errors or warnings, but I still have problems when trying to import any Scipy module that depends on "sparse."  

Build command:

python setup.py config --compiler=intel --fcompiler=intelem install --prefix=/home/cfinch 2>&1 > scipy_build_log.txt

Results:
>>> from scipy import sparse
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/cfinch/lib/python2.5/site-packages/scipy/sparse/__init__.py", line 6, in <module>
    from csr import *
  File "/home/cfinch/lib/python2.5/site-packages/scipy/sparse/csr.py", line 12, in <module>
    from sparsetools import csr_tocsc, csr_tobsr, csr_count_blocks, \
  File "/home/cfinch/lib/python2.5/site-packages/scipy/sparse/sparsetools/__init__.py", line 4, in <module>
    from csr import *
  File "/home/cfinch/lib/python2.5/site-packages/scipy/sparse/sparsetools/csr.py", line 7, in <module>
    import _csr
ImportError: /home/cfinch/lib/python2.5/site-packages/scipy/sparse/sparsetools/_csr.so: undefined symbol: _ZNSt8ios_base4InitD1Ev

When I use nm to look at the symbols in _csr.so, I see that these are undefined, along with several dozen other Python-related symbols:
...
00000000000d2810 W _ZNSt6vectorIxSaIxEE6resizeEmx
00000000000d2fdc W _ZNSt6vectorIySaIyEE6resizeEmy
                 U _ZNSt8ios_base4InitC1Ev
                 U _ZNSt8ios_base4InitD1Ev

I have no idea what this symbol is, or where it's supposed to come from.  Has anyone seen an error like this before?  I am going to see what kind of support I can get from Intel.

   Craig


----- Original Message ----
From: Craig Finch <oanjao@yahoo.com>
To: David Cournapeau <cournape@gmail.com>; SciPy Users List <scipy-user@scipy.org>
Sent: Sunday, July 5, 2009 2:30:18 PM
Subject: Re: [SciPy-user] Scipy weave errors: undefined symbols when importing  compiled extension

By "build log" I assume you mean the output to stderr and stdout when when building Scipy.  It's a huge amount of output, and I'm not sure exactly what to look for.  Do you want me to post a particular section?  Below is the output from sections where g++ is called.  It seems to be linked with g++ everywhere.

One more potential problem area: I'm linking with the Intel Math Kernel Library (MKL) which was almost certainly built with Intel compilers.  Could that be causing this problem?

   Craig
  
-----------------------
building 'scipy.interpolate._interpolate' extension
compiling C++ sources
C compiler: g++ -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -fPIC

compile options: '-Iscipy/interpolate/src -I/home/cfinch/lib/python2.5/site-packages/numpy/core/include -I/home/cfinch/include/python2.5 -c'
g++: scipy/interpolate/src/_interpolate.cpp
g++ -pthread -shared build/temp.linux-x86_64-2.5/scipy/interpolate/src/_interpolate.o -Lbuild/temp.linux-x86_64-2.5 -o build/lib.linux-x86_64-2.5/scipy/interpolate/_interpolate.so
-----------------------
building 'scipy.sparse.sparsetools._csr' extension
compiling C++ sources
C compiler: g++ -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -fPIC

creating build/temp.linux-x86_64-2.5/scipy/sparse/sparsetools
compile options: '-I/home/cfinch/lib/python2.5/site-packages/numpy/core/include -I/home/cfinch/include/python2.5 -c'
g++: scipy/sparse/sparsetools/csr_wrap.cxx
g++ -pthread -shared build/temp.linux-x86_64-2.5/scipy/sparse/sparsetools/csr_wrap.o -Lbuild/temp.linux-x86_64-2.5 -o build/lib.linux-x86_64-2.5/scipy/sparse/sparsetools/_csr.so
building 'scipy.sparse.sparsetools._csc' extension
compiling C++ sources
C compiler: g++ -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -fPIC

compile options: '-I/home/cfinch/lib/python2.5/site-packages/numpy/core/include -I/home/cfinch/include/python2.5 -c'
g++: scipy/sparse/sparsetools/csc_wrap.cxx
g++ -pthread -shared build/temp.linux-x86_64-2.5/scipy/sparse/sparsetools/csc_wrap.o -Lbuild/temp.linux-x86_64-2.5 -o build/lib.linux-x86_64-2.5/scipy/sparse/sparsetools/_csc.so
building 'scipy.sparse.sparsetools._coo' extension
compiling C++ sources
C compiler: g++ -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -fPIC

compile options: '-I/home/cfinch/lib/python2.5/site-packages/numpy/core/include -I/home/cfinch/include/python2.5 -c'
g++: scipy/sparse/sparsetools/coo_wrap.cxx
g++ -pthread -shared build/temp.linux-x86_64-2.5/scipy/sparse/sparsetools/coo_wrap.o -Lbuild/temp.linux-x86_64-2.5 -o build/lib.linux-x86_64-2.5/scipy/sparse/sparsetools/_coo.so
building 'scipy.sparse.sparsetools._bsr' extension
compiling C++ sources
C compiler: g++ -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -fPIC

compile options: '-I/home/cfinch/lib/python2.5/site-packages/numpy/core/include -I/home/cfinch/include/python2.5 -c'
g++: scipy/sparse/sparsetools/bsr_wrap.cxx
g++ -pthread -shared build/temp.linux-x86_64-2.5/scipy/sparse/sparsetools/bsr_wrap.o -Lbuild/temp.linux-x86_64-2.5 -o build/lib.linux-x86_64-2.5/scipy/sparse/sparsetools/_bsr.so
building 'scipy.sparse.sparsetools._dia' extension
compiling C++ sources
C compiler: g++ -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -fPIC

compile options: '-I/home/cfinch/lib/python2.5/site-packages/numpy/core/include -I/home/cfinch/include/python2.5 -c'
g++: scipy/sparse/sparsetools/dia_wrap.cxx
g++ -pthread -shared build/temp.linux-x86_64-2.5/scipy/sparse/sparsetools/dia_wrap.o -Lbuild/temp.linux-x86_64-2.5 -o build/lib.linux-x86_64-2.5/scipy/sparse/sparsetools/_dia.so

-----------------------




----- Original Message ----
From: David Cournapeau <cournape@gmail.com>
To: Craig Finch <cfinch@ieee.org>; SciPy Users List <scipy-user@scipy.org>
Sent: Friday, July 3, 2009 10:21:28 PM
Subject: Re: [SciPy-user] Scipy weave errors: undefined symbols when importing  compiled extension

On Sat, Jul 4, 2009 at 3:38 AM, Craig Finch<oanjao@yahoo.com> wrote:

> ImportError: /home/cfinch/lib/python2.5/site-packages/scipy/sparse/sparsetools/_csr.so: undefined symbol: __gxx_personality_v0
>
> As far as I know, everything Python-related on this machine has now been built with GNU tools, but something is still broken.  One more thing--Python, Numpy, and Scipy are all locally installed in my /home/cfinch directory because the system Python is frozen at 2.4.

This is still a C++ problem, I have sometimes noticed this problem on
some machines. Normally this is caused by a C++ object file being
linked with gcc (instead of g++). Can you post the build log of scipy
?

David


      


More information about the SciPy-user mailing list