[SciPy-dev] distutils, mtrand, Monte Carlo

Ed Schofield schofield at ftw.at
Wed Apr 19 11:08:14 CDT 2006


Pearu Peterson wrote:
> On Wed, 19 Apr 2006, Ed Schofield wrote:
>   
>> I've been playing around with it some more, and it actually seems that
>> config.add_extension already supports building and installing shared
>> libraries.  Normally this is used for building Python extension modules,
>> but the symbol table seems to be independent of Python unless the source
>> files explicitly use Python symbols.  So, unless I'm very mistaken, this
>> works already -- by using the machinery of add_extension rather than
>> add_library.  Here's an example patch (to NumPy) that builds a beautiful
>> shared library for randomkit:
>>
>> Index: numpy/random/setup.py
>> ===================================================================
>> --- numpy/random/setup.py       (revision 2374)
>> +++ numpy/random/setup.py       (working copy)
>> @@ -32,6 +32,10 @@
>>                         )
>>
>>     config.add_data_files(('.', join('mtrand', 'randomkit.h')))
>> +
>> +    config.add_extension('librandomkit',
>> +                         sources=[join('mtrand', 'randomkit.c')],
>> +                         depends=[join('mtrand', 'randomkit.h')])
>>
>>     return config
>>     
>
> Hmm, I haven't thought about the above approach but it can work. However,
> there are some side effect that we should get rid of:
>    the shared library will be linked against python library and so it's
>    an unclean solution
>
> To do things right, we should review command/build_ext.py code and
> see if it can be used for building "clean" shared libraries when
> Extension object has some is_shared_library flag set True (that is set via 
> (to be implemented) add_shared_library method). If that does not work,
> the next step is to copy necessary hooks for shared libraries from
> build_ext.py to build_clib.py or create a new command build_slib (or 
> something like that) altogether.
>   

Okay, here are some more details.  Shared libraries built this way under
Linux look perfectly "clean":

$ ldd
/usr/lib/python2.4/site-packages/numpy/random/librandomkit.so        
linux-gate.so.1 =>  (0xffffe000)
        libpthread.so.0 => /lib/tls/i686/cmov/libpthread.so.0 (0xb7eee000)
        libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0xb7dbf000)
        /lib/ld-linux.so.2 (0x80000000)

and

$ objdump -x librandomkit.so  | grep py

turns up no Python symbols.

Under Win32/MinGW this approach builds a .pyd file.  According to
various distutils list postings, this is just a renamed DLL, but one
that Python expects to export certain symbols.  But librandomkit.pyd
also appears clean of Python symbols, according to my 30-day trial DLL
explorer; the only dependencies are:
    advapi32.dll
    kernel32.dll
    msvcr71.dll
    msvcrt.dll

So it seems the GNU linker does the right thing in the MinGW case too,
not linking in Python symbols unless they're explicitly dereferenced.

I've now got the SciPy montecarlo build working with MinGW too.  It
required three workarounds:

- The first is to rename the randomkit.pyd file in
site-packages\numpy\random\ to randomkit.dll.  This is necessary for the
build; otherwise the linker complains it cannot find -lrandomkit.

- The second is to remove the runtime_library_dirs option from the
config.add_extension call in montecarlo\setup.py.  Keeping this seems to
trigger a weird bug in Python distutils(!) on this platform, where for
some reason sysconfig.get_config_var("CC") returns None.  The function
in question is runtime_library_dir_option() in unixccompiler.py, which
has the comment
    # XXX Hackish, at the very least.  See Python bug #445902

- The third is to copy the randomkit.dll file to
site-packages\scipy\montecarlo\randomkit.pyd.

Of course, manually copying the file defeats the purpose of a dynamic
link library.  This can probably be avoided by using the correct link
flags, or maybe specifying a DLL definition file.

So I agree with your conclusion that making this robustly across
different platforms would require some more work ;)  But it looks
possible ...


-- Ed





More information about the Scipy-dev mailing list