[SciPy-dev] distutils, mtrand, Monte Carlo

[Pearu Peterson]

On Wed, 19 Apr 2006, Ed Schofield wrote:

>> That would be a desirable feature. For example, when scipy would ship
>> blas/lapack source codes then it would be desirable to install blas/lapack
>> libraries so that various scipy subpackages could use them.
>> I have started implementing the support for this feature about half a year
>> ago but I haven't finished it yet, mostly because of I haven't had time
>> to figure out what would be good (portable,etc) locations of such
>> libraries and how would subpackages access them in different situations
>> (building whole scipy, building a subpackage without building scipy,
>> etc..).
>>
> I've been playing around with it some more, and it actually seems that
> config.add_extension already supports building and installing shared
> libraries.  Normally this is used for building Python extension modules,
> but the symbol table seems to be independent of Python unless the source
> files explicitly use Python symbols.  So, unless I'm very mistaken, this
> works already -- by using the machinery of add_extension rather than
> add_library.  Here's an example patch (to NumPy) that builds a beautiful
> shared library for randomkit:
>
> Index: numpy/random/setup.py
> ===================================================================
> --- numpy/random/setup.py       (revision 2374)
> +++ numpy/random/setup.py       (working copy)
> @@ -32,6 +32,10 @@
>                         )
>
>     config.add_data_files(('.', join('mtrand', 'randomkit.h')))
> +
> +    config.add_extension('librandomkit',
> +                         sources=[join('mtrand', 'randomkit.c')],
> +                         depends=[join('mtrand', 'randomkit.h')])
>
>     return config

Hmm, I haven't thought about the above approach but it can work. However,
there are some side effect that we should get rid of:
   the shared library will be linked against python library and so it's
   an unclean solution

To do things right, we should review command/build_ext.py code and
see if it can be used for building "clean" shared libraries when
Extension object has some is_shared_library flag set True (that is set via 
(to be implemented) add_shared_library method). If that does not work,
the next step is to copy necessary hooks for shared libraries from
build_ext.py to build_clib.py or create a new command build_slib (or 
something like that) altogether.


> I've also committed a new sandbox/montecarlo/setup.py file to scipy that
> uses this.  The relevant lines are:
>
>    random_lib_dir = os.path.dirname(numpy.random.__file__)
>
>    config.add_extension('_intsampler',
>              include_dirs = [numpy.get_numpy_include(), random_lib_dir],
>              libraries=['randomkit'],
>              library_dirs=[random_lib_dir],
>              runtime_library_dirs=[random_lib_dir],
>              sources = [join('src', f) for f in
>                        ['_intsamplermodule.c', 'compact5table.c']])
>
> This works fine under Linux.  I'll also test it out with Win32/MinGW.
> (By the way, does distutils provide its own way of retrieving the
> site-packages/numpy/random directory?)

See

     distutils.sysconfig.get_python_lib

function. Using smth like

   random_lib_dir = os.path.join(get_python_lib(),'numpy','random')

should work in this particular case but it's not a general solution. For 
in-place builds random_lib_dir should probably be the one starting with
path that build_py.get_package_dir(..) method would return, for instance.

So, supporting installing shared libraries in a robust way may require 
some work. Though, simple solutions like above might work already for 
certain platforms, we should be careful for not breaking scipy/numpy 
builds for other platforms.

Pearu