[SciPy-User] scipy import problem in apache-mod_wsgi environment

Myunghwa Hwang mhwang4@gmail....
Wed Nov 16 19:39:39 CST 2011


Hi, Hayne, Charles, and Bruce!

Thanks for your response.
I was out of town and got into other projects.

For Hayne's question, when I included only the two lines including the
finfo function, my application hung.
For Charles's question, all computer nodes in our server use centos 6 linux
system. For hardware specifics, I know only they are all 64-bit systems.
For Bruce's question, I am aware of the selinux problem. It was not the
issue.

Actually, my problem seemed related to how mod_wsgi handles multiple python
processes.
My problem was solved by including an apache directive related to mod_wsgi.
Particularly, I added the following line into my apache configuration file:
WSGIApplicationGroup %{GLOBAL}

Here is the explanation of this apache directive:

The application group name will be set to the empty string.

Any WSGI applications in the global application group will always be
executed within the context of the first interpreter created by Python when
it is initialised. Forcing a WSGI application to run within the first
interpreter can be necessary when a third party C extension module for
Python has used the simplified threading API for manipulation of the Python
GIL and thus will not run correctly within any additional sub interpreters
created by Python.

Still, I am not quite sure about the real source that caused my problem.
But, it is gone now. This is all thanks to you and scipy mailing list.
Again, I really appreciate your help.

Myunghwa

On Mon, Nov 7, 2011 at 11:47 AM, Ralf Gommers
<ralf.gommers@googlemail.com>wrote:

>
>
> On Mon, Nov 7, 2011 at 5:43 AM, Myunghwa Hwang <mhwang4@gmail.com> wrote:
>
>> Hi, Hayne!
>>
>> Thanks for your answer.
>> After trying out what you suggested (that is, commenting out the import
>> of decomp),
>> I found out the import of decomp was not the problem.
>> In decomp_schur, there are two lines checking something related to
>> rounding errors specific to a single machine as follows:
>> eps = np.finfo(float).eps
>> feps = numpy.finfo(single).eps
>>
>> If you execute just the above lines in your application instead of
> importing scipy, does it hang too?
>
> Ralf
>
>
>
>>  When scipy reaches the above lines, my application hangs.
>> I found a web document where the author encountered the same problem with
>> these lines but in different contexts:
>>
>> http://stackoverflow.com/questions/7592565/when-embedding-cpython-in-java-why-does-this-hang
>>
>> The discussion in the web document is not applicable to my problem.
>> Also, the np.finfo statements seem to exist in multiple modules of scipy.
>> Without addressing all related modules manually,
>> would it be any other solutions?
>>
>> Thanks!
>>
>> --Myung-Hwa
>>
>>
>> On Sat, Nov 5, 2011 at 1:14 PM, <hayne@sympatico.ca> wrote:
>>
>>> I would try putting print statements inside "decomp_schur.py" since that
>>> is the module that you said is causing problems.
>>> Print out the contents of the dictionary sys.modules just before the
>>> import of decomp in "decomp_schur.py". Is 'decomp' in the dictionary?
>>> What happens if you comment-out the import of decomp in
>>> "decomp_schur.py"  ?
>>> --
>>> Cameron Hayne
>>> macdev@hayne.net
>>>
>>>
>>>
>>> On 5-Nov-11, at 3:59 PM, Myunghwa Hwang wrote:
>>>
>>>> I am trying to run a simple django application in a cluster environment.
>>>> And, my application hangs while it imports scipy.linalg, and both scipy
>>>> and apache do not write out error messages.
>>>> When I run my application in my local python shell, it imports
>>>> scipy.linalg. But, somehow it does not when it is run by apache.
>>>> So, after reading this message, please share any ideas about how to
>>>> debug this problem or new solutions to address this issue or deploy my
>>>> application.
>>>>
>>>> Now, let me explain our current setup.
>>>> 1. OS
>>>> -- The server is a compute cluster where each node runs centos 6 that
>>>> was installed from a clean version of centos6 minimal.2. Apache
>>>>
>>>> -- Apache 2.2 was also manually installed from one of default linux
>>>> repository.  To be specific, it was installed from its source code together
>>>> with httpd-dev.
>>>> 3. Python
>>>> -- Python 2.7.2 was also installed from its source code across all
>>>> nodes in the cluster. Its source code was downloaded from python.org's
>>>> ftp.
>>>> 4. Python packages: nose, numpy, scipy
>>>> -- Nose 1.1.2 was downloaded from pypi.python.org and installed from
>>>> its source code.
>>>> -- numpy 1.6.1 was downloaded and installed from a linux repository.
>>>> When building numpy, gnu95 fortran complier was used.
>>>> -- To install scipy, we installed atlas-3.8.4, lapack-3.3.1, and blas
>>>> from their source code.----- atlas was from sourceforge's 3.8.4 stable
>>>> version. To compile altas, gcc was used.
>>>>
>>>> ----- lapack and blas was obtained from netlib.org's repository. To
>>>> compile the package of lapack and blas, gforan was used.
>>>> ----- Finally, after exporting paths to blas, lapack, and atlas,
>>>> scipy-0.9.0 was installed from its source code.
>>>>      scipy was obtained from sourceforge.net's repository.
>>>> A note that contains the above information about software installation
>>>> is attached.
>>>>
>>>> All of the above were installed in the same way across all nodes in our
>>>> cluster.
>>>> Since I am the only user of the cluster who needs to run python web
>>>> applications,
>>>> I installed python virtualenv package in my local directory.
>>>> Within my virtual environment, django-1.3 and pysal-1.2 (our own
>>>> package) were installed.
>>>> To deploy my web applications, we used mod_wsgi.
>>>> mod-wsgi was compiled with python-2.7.2 and loaded into apache-2.2.
>>>> My application is attached. Basically, it is a 'hello world'
>>>> application that tests if numpy, scipy, and pysal can be imported.
>>>> In the attached file, lines 4-9 are just adding paths to django and
>>>> pysal so that apache knows where to find these packages.
>>>> Also, to let apache know where to find atlas-related packages, the path
>>>> to those packages was added to the LD_LIBRARY_PATH environment variable in
>>>> the /etc/sysconfig/httpd file.
>>>>
>>>> When I first ran my application, it just hung and wrote no message.
>>>> So, across scipy.linalg modules, I added print out statements to figure
>>>> out at which point the import was broken.
>>>> Here is the messages I got when I imported scipy.linalg in my local
>>>> python shell.
>>>>        • ########################
>>>>        • starting linalg.__init__
>>>>        • pre __init__.__doc__
>>>>        • pre __init__.__version__
>>>>        • pre __init__.misc
>>>>        • pre __init__.basic
>>>>        • #######################
>>>>        • Starting basic
>>>>        • pre basic.flinalg
>>>>        • pre basic.lapack
>>>>        • pre basic.misc
>>>>        • pre basic.scipy.linalg
>>>>        • pre basic.decomp_svd
>>>>        • pre __init__.decomp
>>>>        • ################
>>>>        • starting decomp
>>>>        • pre decomp.array et al.
>>>>        • pre decomp.calc_lwork
>>>>        • pre decomp.LinAlgError
>>>>        • pre decomp.get_lapack_funcs
>>>>        • pre decomp.get_blas_funcs
>>>>        • ####################
>>>>        • Starting blas
>>>>        • pre blas.scipy.linalg.fblas
>>>>        • pre blas.scipy.linalg.cblas
>>>>        • pre __init__.decomp_lu
>>>>        • pre __init__.decomp_cholesky
>>>>        • pre __init__.decomp_qr
>>>>        • #################
>>>>        • Starting special_matrices
>>>>        • pre special_matrices.math
>>>>        • pre special_matrices.np
>>>>        • pre __init__.decomp_svd
>>>>        • pre __init__.decomp_schur
>>>>        • ##################
>>>>        • starting schur...
>>>>        • pre decomp_schur.misc
>>>>        • pre decomp_schur.LinAlgError
>>>>        • pre decomp_schur.get_lapack_funcs
>>>>        • pre decomp_schur.eigvals:**1320454147.23Fri Nov  4 17:49:07
>>>> 2011
>>>>        • schur testing
>>>>        • pre __init__.matfuncs
>>>>        • #####################
>>>>        • Starting matfuncs
>>>>        • pre matfuncs. asarray et al
>>>>        • pre matfuncs.matrix
>>>>        • pre matfuncs.np
>>>>        • pre matfuncs.misc
>>>>        • pre matfuncs.basic
>>>>        • pre matfuncs.special_matrices
>>>>        • pre matfuncs.decomp
>>>>        • pre matfuncs.decomp_svd
>>>>        • pre matfuncs.decomp_schur
>>>>        • pre __init__.blas
>>>>        • pre __init__.special_matrices
>>>> When scipy.linalg is successfully imported, I should get these messages.
>>>> But, when my web application tried to import scipy.linalg, the output
>>>> messages stop at line 41.
>>>> At line 41, decomp_schur.py tries to import decomp.py. Since decomp.py
>>>> was already imported at line 16, scipy ignores it and continues to import
>>>> other modules in my local shell.
>>>> But, somehow, in apache-mod_wsgi environment, scipy failed to ignore or
>>>> reload decomp.py and seems to kill my web application.
>>>> This is really odd, because python does not give any message about this
>>>> error and neither does apache. apache just hangs without sending out any
>>>> response.
>>>> Since lapack and blas functions were imported successfully, the problem
>>>> seems not related to path setup.
>>>>
>>>> If anyone in the list has any insights into or experience into this
>>>> kind of symptom,
>>>> please share your insights and experience. In particular, debugging
>>>> techniques or less-known installation/compilation problems would be helpful.
>>>> I feel like I am at a dead end. So, please help me.
>>>>
>>>> Thanks for reading this post.
>>>> I will look forward to your responses.
>>>>
>>>> -- Myung-Hwa Hwang
>>>>
>>>> --
>>>> Myung-Hwa Hwang
>>>> GeoDa Center
>>>> School of Geographical Sciences and Urban Planning
>>>> Arizona State University
>>>> mhwang4@gmail.com or Myunghwa.Hwang@asu.edu
>>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
>> --
>> Myung-Hwa Hwang
>> GeoDa Center
>> School of Geographical Sciences and Urban Planning
>> Arizona State University
>> mhwang4@gmail.com or Myunghwa.Hwang@asu.edu
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User@scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User@scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>


-- 
Myung-Hwa Hwang
GeoDa Center
School of Geographical Sciences and Urban Planning
Arizona State University
mhwang4@gmail.com or Myunghwa.Hwang@asu.edu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/scipy-user/attachments/20111116/e85117d3/attachment-0001.html 


More information about the SciPy-User mailing list