[SciPy-User] scipy import problem in apache-mod_wsgi environment

Ralf Gommers ralf.gommers@googlemail....
Mon Nov 7 12:47:28 CST 2011


On Mon, Nov 7, 2011 at 5:43 AM, Myunghwa Hwang <mhwang4@gmail.com> wrote:

> Hi, Hayne!
>
> Thanks for your answer.
> After trying out what you suggested (that is, commenting out the import of
> decomp),
> I found out the import of decomp was not the problem.
> In decomp_schur, there are two lines checking something related to
> rounding errors specific to a single machine as follows:
> eps = np.finfo(float).eps
> feps = numpy.finfo(single).eps
>
> If you execute just the above lines in your application instead of
importing scipy, does it hang too?

Ralf



> When scipy reaches the above lines, my application hangs.
> I found a web document where the author encountered the same problem with
> these lines but in different contexts:
>
> http://stackoverflow.com/questions/7592565/when-embedding-cpython-in-java-why-does-this-hang
>
> The discussion in the web document is not applicable to my problem.
> Also, the np.finfo statements seem to exist in multiple modules of scipy.
> Without addressing all related modules manually,
> would it be any other solutions?
>
> Thanks!
>
> --Myung-Hwa
>
>
> On Sat, Nov 5, 2011 at 1:14 PM, <hayne@sympatico.ca> wrote:
>
>> I would try putting print statements inside "decomp_schur.py" since that
>> is the module that you said is causing problems.
>> Print out the contents of the dictionary sys.modules just before the
>> import of decomp in "decomp_schur.py". Is 'decomp' in the dictionary?
>> What happens if you comment-out the import of decomp in "decomp_schur.py"
>>  ?
>> --
>> Cameron Hayne
>> macdev@hayne.net
>>
>>
>>
>> On 5-Nov-11, at 3:59 PM, Myunghwa Hwang wrote:
>>
>>> I am trying to run a simple django application in a cluster environment.
>>> And, my application hangs while it imports scipy.linalg, and both scipy
>>> and apache do not write out error messages.
>>> When I run my application in my local python shell, it imports
>>> scipy.linalg. But, somehow it does not when it is run by apache.
>>> So, after reading this message, please share any ideas about how to
>>> debug this problem or new solutions to address this issue or deploy my
>>> application.
>>>
>>> Now, let me explain our current setup.
>>> 1. OS
>>> -- The server is a compute cluster where each node runs centos 6 that
>>> was installed from a clean version of centos6 minimal.2. Apache
>>>
>>> -- Apache 2.2 was also manually installed from one of default linux
>>> repository.  To be specific, it was installed from its source code together
>>> with httpd-dev.
>>> 3. Python
>>> -- Python 2.7.2 was also installed from its source code across all nodes
>>> in the cluster. Its source code was downloaded from python.org's ftp.
>>> 4. Python packages: nose, numpy, scipy
>>> -- Nose 1.1.2 was downloaded from pypi.python.org and installed from
>>> its source code.
>>> -- numpy 1.6.1 was downloaded and installed from a linux repository.
>>> When building numpy, gnu95 fortran complier was used.
>>> -- To install scipy, we installed atlas-3.8.4, lapack-3.3.1, and blas
>>> from their source code.----- atlas was from sourceforge's 3.8.4 stable
>>> version. To compile altas, gcc was used.
>>>
>>> ----- lapack and blas was obtained from netlib.org's repository. To
>>> compile the package of lapack and blas, gforan was used.
>>> ----- Finally, after exporting paths to blas, lapack, and atlas,
>>> scipy-0.9.0 was installed from its source code.
>>>      scipy was obtained from sourceforge.net's repository.
>>> A note that contains the above information about software installation
>>> is attached.
>>>
>>> All of the above were installed in the same way across all nodes in our
>>> cluster.
>>> Since I am the only user of the cluster who needs to run python web
>>> applications,
>>> I installed python virtualenv package in my local directory.
>>> Within my virtual environment, django-1.3 and pysal-1.2 (our own
>>> package) were installed.
>>> To deploy my web applications, we used mod_wsgi.
>>> mod-wsgi was compiled with python-2.7.2 and loaded into apache-2.2.
>>> My application is attached. Basically, it is a 'hello world' application
>>> that tests if numpy, scipy, and pysal can be imported.
>>> In the attached file, lines 4-9 are just adding paths to django and
>>> pysal so that apache knows where to find these packages.
>>> Also, to let apache know where to find atlas-related packages, the path
>>> to those packages was added to the LD_LIBRARY_PATH environment variable in
>>> the /etc/sysconfig/httpd file.
>>>
>>> When I first ran my application, it just hung and wrote no message.
>>> So, across scipy.linalg modules, I added print out statements to figure
>>> out at which point the import was broken.
>>> Here is the messages I got when I imported scipy.linalg in my local
>>> python shell.
>>>        • ########################
>>>        • starting linalg.__init__
>>>        • pre __init__.__doc__
>>>        • pre __init__.__version__
>>>        • pre __init__.misc
>>>        • pre __init__.basic
>>>        • #######################
>>>        • Starting basic
>>>        • pre basic.flinalg
>>>        • pre basic.lapack
>>>        • pre basic.misc
>>>        • pre basic.scipy.linalg
>>>        • pre basic.decomp_svd
>>>        • pre __init__.decomp
>>>        • ################
>>>        • starting decomp
>>>        • pre decomp.array et al.
>>>        • pre decomp.calc_lwork
>>>        • pre decomp.LinAlgError
>>>        • pre decomp.get_lapack_funcs
>>>        • pre decomp.get_blas_funcs
>>>        • ####################
>>>        • Starting blas
>>>        • pre blas.scipy.linalg.fblas
>>>        • pre blas.scipy.linalg.cblas
>>>        • pre __init__.decomp_lu
>>>        • pre __init__.decomp_cholesky
>>>        • pre __init__.decomp_qr
>>>        • #################
>>>        • Starting special_matrices
>>>        • pre special_matrices.math
>>>        • pre special_matrices.np
>>>        • pre __init__.decomp_svd
>>>        • pre __init__.decomp_schur
>>>        • ##################
>>>        • starting schur...
>>>        • pre decomp_schur.misc
>>>        • pre decomp_schur.LinAlgError
>>>        • pre decomp_schur.get_lapack_funcs
>>>        • pre decomp_schur.eigvals:**1320454147.23Fri Nov  4 17:49:07
>>> 2011
>>>        • schur testing
>>>        • pre __init__.matfuncs
>>>        • #####################
>>>        • Starting matfuncs
>>>        • pre matfuncs. asarray et al
>>>        • pre matfuncs.matrix
>>>        • pre matfuncs.np
>>>        • pre matfuncs.misc
>>>        • pre matfuncs.basic
>>>        • pre matfuncs.special_matrices
>>>        • pre matfuncs.decomp
>>>        • pre matfuncs.decomp_svd
>>>        • pre matfuncs.decomp_schur
>>>        • pre __init__.blas
>>>        • pre __init__.special_matrices
>>> When scipy.linalg is successfully imported, I should get these messages.
>>> But, when my web application tried to import scipy.linalg, the output
>>> messages stop at line 41.
>>> At line 41, decomp_schur.py tries to import decomp.py. Since decomp.py
>>> was already imported at line 16, scipy ignores it and continues to import
>>> other modules in my local shell.
>>> But, somehow, in apache-mod_wsgi environment, scipy failed to ignore or
>>> reload decomp.py and seems to kill my web application.
>>> This is really odd, because python does not give any message about this
>>> error and neither does apache. apache just hangs without sending out any
>>> response.
>>> Since lapack and blas functions were imported successfully, the problem
>>> seems not related to path setup.
>>>
>>> If anyone in the list has any insights into or experience into this kind
>>> of symptom,
>>> please share your insights and experience. In particular, debugging
>>> techniques or less-known installation/compilation problems would be helpful.
>>> I feel like I am at a dead end. So, please help me.
>>>
>>> Thanks for reading this post.
>>> I will look forward to your responses.
>>>
>>> -- Myung-Hwa Hwang
>>>
>>> --
>>> Myung-Hwa Hwang
>>> GeoDa Center
>>> School of Geographical Sciences and Urban Planning
>>> Arizona State University
>>> mhwang4@gmail.com or Myunghwa.Hwang@asu.edu
>>>
>>
>>
>>
>>
>>
>
>
> --
> Myung-Hwa Hwang
> GeoDa Center
> School of Geographical Sciences and Urban Planning
> Arizona State University
> mhwang4@gmail.com or Myunghwa.Hwang@asu.edu
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User@scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/scipy-user/attachments/20111107/d3904ddb/attachment.html 


More information about the SciPy-User mailing list