[SciPy-User] scipy import problem in apache-mod_wsgi environment

Myunghwa Hwang mhwang4@gmail....
Sun Nov 6 22:51:32 CST 2011


Ralf and Bruce,

Thanks for your answer.
I just found that the problem area is the following lines:
eps = np.finfo(float).eps
feps = np.finfo(single).eps

When scipy hits these lines, my app hangs.
Do you have any ideas about why these lines raise the issue?

Thanks,

Myung-Hwa

On Sun, Nov 6, 2011 at 2:49 PM, Ralf Gommers <ralf.gommers@googlemail.com>wrote:

>
>
> On Sat, Nov 5, 2011 at 8:52 PM, Myunghwa Hwang <mhwang4@gmail.com> wrote:
>
>> Hello, list!
>>
>> First of all, this will be a long message, due to the complexity of our
>> environment.
>> So, please be patient with my question.
>>
>> I am trying to run a simple django application in a cluster environment.
>> And, my application hangs while it imports scipy.linalg, and both scipy
>> and apache do not write out error messages.
>> When I run my application in my local python shell, it imports
>> scipy.linalg. But, somehow it does not when it is run by apache.
>> So, after reading this message, please share any ideas about how to debug
>> this problem or new solutions to address this issue or deploy my
>> application.
>>
>> Now, let me explain our current setup.
>> 1. OS
>> -- The server is a compute cluster where each node runs centos 6 that was
>> installed from a clean version of centos6 minimal.
>> 2. Apache
>> -- Apache 2.2 is also manually installed from one of default linux
>> repository.
>>   To be specific, it is installed from source code together with
>> httpd-dev.
>> 3. Python
>> -- Python 2.7.2 is also installed from source code across all nodes in
>> the cluster. Its source code is downloaded from python.org's ftp.
>> 4. Python packages: nose, numpy, scipy
>> -- Nose 1.1.2 was downloaded from pypi.python.org and installed from its
>> source code.
>> -- numpy 1.6.1 was downloaded and installed from a linux repository. When
>> building numpy, gnu95 fortran complier was used.
>> -- To install scipy, we installed atlas-3.8.4, lapack-3.3.1, and blas
>> from their source code.
>> ----- atlas was from sourceforge's 3.8.4 stable version. To compile
>> altas, gcc was used.
>> ----- lapack and blas was obtained from netlib.org's repository. To
>> compile the package of lapack and blas, gforan was used.
>> ----- Finally, after exporting paths to blas, lapack, and atlas,
>> scipy-0.9.0 was installed from its source code.
>>       scipy was obtained from sourceforge.net's repository.
>>
>> All of the above were installed in the same way across all nodes in our
>> cluster.
>> Since I am the only user of the cluster who needs to run python web
>> applications,
>> I installed python virtualenv package in my local directory.
>> Within my virtual environment, django-1.3 and pysal-1.2 (our own package)
>> were installed.
>> To deploy my web applications, we used mod_wsgi.
>> mod-wsgi was compiled with python-2.7.2 and loaded into apache-2.2.
>> My application is attached. Basically, it is 'hello world' application
>> that tests if numpy, scipy, and pysal can be imported.
>> In the attached file, lines 4-9 are just adding paths to django and pysal
>> so that apache knows where to find these packages.
>> Also, to let apache know where to find atlas-related packages, the path
>> to those packages was added to the LD_LIBRARY_PATH environment variable in
>> the /etc/sysconfig/httpd file.
>>
>> When I first ran my application, it just hung and wrote no message.
>> So, across scipy.linalg modules, I added print out statements to figure
>> out at which point the import was broken.
>> Here is the messages I got when I imported scipy.linalg in my local
>> python shell.
>>
>>    1. ########################
>>    2. starting linalg.__init__
>>    3. pre __init__.__doc__
>>    4. pre __init__.__version__
>>    5. pre __init__.misc
>>    6. pre __init__.basic
>>    7. #######################
>>    8. Starting basic
>>    9. pre basic.flinalg
>>    10. pre basic.lapack
>>    11. pre basic.misc
>>    12. pre basic.scipy.linalg
>>    13. pre basic.decomp_svd
>>    14. pre __init__.decomp
>>    15. ################
>>    16. starting decomp
>>    17. pre decomp.array et al.
>>    18. pre decomp.calc_lwork
>>    19. pre decomp.LinAlgError
>>    20. pre decomp.get_lapack_funcs
>>    21. pre decomp.get_blas_funcs
>>    22. ####################
>>    23. Starting blas
>>    24. pre blas.scipy.linalg.fblas
>>    25. pre blas.scipy.linalg.cblas
>>    26. pre __init__.decomp_lu
>>    27. pre __init__.decomp_cholesky
>>    28. pre __init__.decomp_qr
>>    29. #################
>>    30. Starting special_matrices
>>    31. pre special_matrices.math
>>    32. pre special_matrices.np
>>    33. pre __init__.decomp_svd
>>    34. pre __init__.decomp_schur
>>    35. ##################
>>    36. starting schur...
>>    37. pre decomp_schur.misc
>>    38. pre decomp_schur.LinAlgError
>>    39. pre decomp_schur.get_lapack_funcs
>>    40. pre decomp_schur.eigvals:1320454147.23Fri Nov  4 17:49:07 2011
>>    41. schur testing
>>    42. pre __init__.matfuncs
>>    43. #####################
>>    44. Starting matfuncs
>>    45. pre matfuncs. asarray et al
>>    46. pre matfuncs.matrix
>>    47. pre matfuncs.np
>>    48. pre matfuncs.misc
>>    49. pre matfuncs.basic
>>    50. pre matfuncs.special_matrices
>>    51. pre matfuncs.decomp
>>    52. pre matfuncs.decomp_svd
>>    53. pre matfuncs.decomp_schur
>>    54. pre __init__.blas
>>    55. pre __init__.special_matrices
>>
>> When scipy.linalg is successfully imported, I should get these messages.
>> But, when my web application tried to import scipy.linalg, the output
>> messages stop at line 41.
>> At line 41, decomp_schur.py tries to import decomp.py. Since decomp.py
>> was already imported at line 16, scipy ignores it and continues to import
>> other modules in my local shell.
>> But, somehow, in apache-mod_wsgi environment, scipy failed to ignore or
>> reload decomp.py and seems to kill my web application.
>> This is really odd, because python does not give any message about this
>> error and neither does apache. apache just hangs without sending out any
>> response.
>> Since lapack and blas functions were imported successfully, the problem
>> seems not related to path setup.
>>
>> If anyone in the list has any insights into or experience into this kind
>> of symptom,
>> please share your insights and experience. In particular, debugging
>> techniques or less-known installation/compilation problems would be helpful.
>> I feel like I am at a dead end. So, please help me.
>>
>> Thanks for reading this post.
>> I will look forward to yo
>>
>
> Looking at linalg/__init__.py the register_func calls are an obvious
> candidate for causing this strange issue. of the few functions left in
> decomp_schur.py, norm() and dot() have both gone through register_func,
> which messes with the call stack. Could you comment out all calls to that
> function and see if that helps? Replace functions that then become
> unavailable with numpy ones of the same name where needed.
>
> Cheers,
> Ralf
>
>
>
>>
>>
>> --
>> Myung-Hwa Hwang
>> GeoDa Center
>> School of Geographical Sciences and Urban Planning
>> Arizona State University
>> mhwang4@gmail.com or Myunghwa.Hwang@asu.edu
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User@scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User@scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>


-- 
Myung-Hwa Hwang
GeoDa Center
School of Geographical Sciences and Urban Planning
Arizona State University
mhwang4@gmail.com or Myunghwa.Hwang@asu.edu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/scipy-user/attachments/20111106/980ad0b9/attachment.html 


More information about the SciPy-User mailing list