[SciPy-User] scipy import problem in apache-mod_wsgi environment

Bruce Southey bsouthey@gmail....
Sun Nov 6 20:56:31 CST 2011


On Sun, Nov 6, 2011 at 3:49 PM, Ralf Gommers
<ralf.gommers@googlemail.com> wrote:
>
>
> On Sat, Nov 5, 2011 at 8:52 PM, Myunghwa Hwang <mhwang4@gmail.com> wrote:
>>
>> Hello, list!
>> First of all, this will be a long message, due to the complexity of our
>> environment.
>> So, please be patient with my question.
>> I am trying to run a simple django application in a cluster environment.
>> And, my application hangs while it imports scipy.linalg, and both scipy
>> and apache do not write out error messages.
>> When I run my application in my local python shell, it imports
>> scipy.linalg. But, somehow it does not when it is run by apache.
>> So, after reading this message, please share any ideas about how to debug
>> this problem or new solutions to address this issue or deploy my
>> application.
>> Now, let me explain our current setup.
>> 1. OS
>> -- The server is a compute cluster where each node runs centos 6 that was
>> installed from a clean version of centos6 minimal.
>> 2. Apache
>> -- Apache 2.2 is also manually installed from one of default linux
>> repository.
>>   To be specific, it is installed from source code together with
>> httpd-dev.
>> 3. Python
>> -- Python 2.7.2 is also installed from source code across all nodes in the
>> cluster. Its source code is downloaded from python.org's ftp.
>> 4. Python packages: nose, numpy, scipy
>> -- Nose 1.1.2 was downloaded from pypi.python.org and installed from its
>> source code.
>> -- numpy 1.6.1 was downloaded and installed from a linux repository. When
>> building numpy, gnu95 fortran complier was used.
>> -- To install scipy, we installed atlas-3.8.4, lapack-3.3.1, and blas from
>> their source code.
>> ----- atlas was from sourceforge's 3.8.4 stable version. To compile altas,
>> gcc was used.
>> ----- lapack and blas was obtained from netlib.org's repository. To
>> compile the package of lapack and blas, gforan was used.
>> ----- Finally, after exporting paths to blas, lapack, and atlas,
>> scipy-0.9.0 was installed from its source code.
>>       scipy was obtained from sourceforge.net's repository.
>> All of the above were installed in the same way across all nodes in our
>> cluster.
>> Since I am the only user of the cluster who needs to run python web
>> applications,
>> I installed python virtualenv package in my local directory.
>> Within my virtual environment, django-1.3 and pysal-1.2 (our own package)
>> were installed.
>> To deploy my web applications, we used mod_wsgi.
>> mod-wsgi was compiled with python-2.7.2 and loaded into apache-2.2.
>> My application is attached. Basically, it is 'hello world' application
>> that tests if numpy, scipy, and pysal can be imported.
>> In the attached file, lines 4-9 are just adding paths to django and pysal
>> so that apache knows where to find these packages.
>> Also, to let apache know where to find atlas-related packages, the path to
>> those packages was added to the LD_LIBRARY_PATH environment variable in the
>> /etc/sysconfig/httpd file.
>> When I first ran my application, it just hung and wrote no message.
>> So, across scipy.linalg modules, I added print out statements to figure
>> out at which point the import was broken.
>> Here is the messages I got when I imported scipy.linalg in my local python
>> shell.
>>
>> ########################
>> starting linalg.__init__
>> pre __init__.__doc__
>> pre __init__.__version__
>> pre __init__.misc
>> pre __init__.basic
>> #######################
>> Starting basic
>> pre basic.flinalg
>> pre basic.lapack
>> pre basic.misc
>> pre basic.scipy.linalg
>> pre basic.decomp_svd
>> pre __init__.decomp
>> ################
>> starting decomp
>> pre decomp.array et al.
>> pre decomp.calc_lwork
>> pre decomp.LinAlgError
>> pre decomp.get_lapack_funcs
>> pre decomp.get_blas_funcs
>> ####################
>> Starting blas
>> pre blas.scipy.linalg.fblas
>> pre blas.scipy.linalg.cblas
>> pre __init__.decomp_lu
>> pre __init__.decomp_cholesky
>> pre __init__.decomp_qr
>> #################
>> Starting special_matrices
>> pre special_matrices.math
>> pre special_matrices.np
>> pre __init__.decomp_svd
>> pre __init__.decomp_schur
>> ##################
>> starting schur...
>> pre decomp_schur.misc
>> pre decomp_schur.LinAlgError
>> pre decomp_schur.get_lapack_funcs
>> pre decomp_schur.eigvals:1320454147.23Fri Nov  4 17:49:07 2011
>> schur testing
>> pre __init__.matfuncs
>> #####################
>> Starting matfuncs
>> pre matfuncs. asarray et al
>> pre matfuncs.matrix
>> pre matfuncs.np
>> pre matfuncs.misc
>> pre matfuncs.basic
>> pre matfuncs.special_matrices
>> pre matfuncs.decomp
>> pre matfuncs.decomp_svd
>> pre matfuncs.decomp_schur
>> pre __init__.blas
>> pre __init__.special_matrices
>>
>> When scipy.linalg is successfully imported, I should get these messages.
>> But, when my web application tried to import scipy.linalg, the output
>> messages stop at line 41.
>> At line 41, decomp_schur.py tries to import decomp.py. Since decomp.py was
>> already imported at line 16, scipy ignores it and continues to import other
>> modules in my local shell.
>> But, somehow, in apache-mod_wsgi environment, scipy failed to ignore or
>> reload decomp.py and seems to kill my web application.
>> This is really odd, because python does not give any message about this
>> error and neither does apache. apache just hangs without sending out any
>> response.
>> Since lapack and blas functions were imported successfully, the problem
>> seems not related to path setup.
>> If anyone in the list has any insights into or experience into this kind
>> of symptom,
>> please share your insights and experience. In particular, debugging
>> techniques or less-known installation/compilation problems would be helpful.
>> I feel like I am at a dead end. So, please help me.
>> Thanks for reading this post.
>> I will look forward to yo
>
> Looking at linalg/__init__.py the register_func calls are an obvious
> candidate for causing this strange issue. of the few functions left in
> decomp_schur.py, norm() and dot() have both gone through register_func,
> which messes with the call stack. Could you comment out all calls to that
> function and see if that helps? Replace functions that then become
> unavailable with numpy ones of the same name where needed.
>
> Cheers,
> Ralf
>
>
>>
>>
>> --
>> Myung-Hwa Hwang
>> GeoDa Center
>> School of Geographical Sciences and Urban Planning
>> Arizona State University
>> mhwang4@gmail.com or Myunghwa.Hwang@asu.edu
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User@scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>
Also, if you provide some installation instruction, I might try to run it.

But I do wonder if this is due to your virtual environment and
associated paths once called via Apache.

Bruce


More information about the SciPy-User mailing list