[Numpy-discussion] NumPy, shared libraries and ctypes

Andrew Straw strawman at astraw.com
Tue Aug 8 20:52:12 CDT 2006

Dear Albert,

I have started to use numpy and ctypes together and I've been quite
pleased. Thanks for your efforts and writings on the wiki.

On the topic of ctypes but not directly following from your email: I
noticed immediately that the .ctypes attribute of an array is going to
be a de-facto array interface, and wondered whether it would actually be
better to write some code that takes the __array_struct__ interface and
exposes that as an object with ctypes-providing attributes. This way, it
could be used by all software exposing the __array_struct__ interface.
Still, even with today's implementation, this could be acheived with
numpy.asarray( my_array_struct_object ).ctypes.

Back to your email: I don't understand why you're trying to build a
shared library with distutils. What's wrong with a plain old c-compiler
and linker (and mt.exe if you're using MS VC 8)? You can build shared
libraries this way with Makefiles, scons, Visual Studio, and about a
billion other solutions that have evolved since early C days. I can
understand the desire of getting "python setup.py install" to work, but
I suspect spawning an appropriate subprocess to do the compilation would
be easier and more robust than attempting to get distutils to do
something it's not designed for.  (Then again, to see what numpy
distutils can do, well, let's just say I'm amazed.) Along these lines, I
noticed that ctypes-itself seems to have put some hooks into setup.py to
perform at least part of the configure/make dance on linux, although I
haven't investigated any further yet. Perhaps that's a better way to go
than bending distutils to your will?

Finally, the ctypes_load_library() function was broken for me and so I
just ended up using the appropriate ctypes calls directly. (I should
report this bug, I know, and I haven't yet... Bad Andrew.) But the
bigger issue for me is that this is a ctypes-level convenience function,
and I can't see why it should be in numpy. Is there any reason it should
go in numpy and not into ctypes itself where it would surely receive
more review and widespread use if it's useful?

Albert Strasheim wrote:

>Hello all
>With the nice ctypes integration in NumPy, and with Python 2.5 which will
>include ctypes around the corner, a remote possibility exists that within
>the next year or two, I might not be the only person that wants to use NumPy
>with ctypes.
>This is probably going to mean that this someone is going to want to build a
>shared library for use with ctypes. This is all well and good if you're
>using a build tool that knows about shared libraries, but in case this
>person is stuck with distutils, here is what we might want to do.
>Following this thread from SciPy-dev:
>I came up with the following plan.
>As it happens, pretending your shared library is a Python extension mostly
>works. In your setup.py you can do something like this:
>config = Configuration(package_name,parent_package,top_path)
>                     define_macros=[('LIBSVM_EXPORTS', None),
>                                    ('LIBSVM_DLL', None)],
>                     sources=[join('libsvm-2.82', 'svm.cpp')],
>                     depends=[join('libsvm-2.82', 'svm.h')])
>First caveat: on Windows, distutils forces the linker to look for an
>exported symbol called init<yourextensionname>. In your code you'll have to
>add an empty function like this:
>void initlibsvm_() {}
>This gets us a compiled Python extension, which also happens to be a shared
>library on every platform I know of, which is Linux and Windows.
>Counter-examples anyone?.
>Next caveat: on Windows, shared libraries aka DLLs, typically have a .dll
>extension. However, Python extensions have a .pyd extension.
>We have a utility function in NumPy called ctypes_load_library which handles
>finding and loading of shared libraries with ctypes. Currently, shared
>library extensions (.dll, .so, .dylib) are hardcoded in this function.
>I propose we modify this function to look something like this:
>def ctypes_load_library(libname, loader_path, distutils_hack=False):
>    ...
>If distutils_hack is True, instead of the default mechanism (which is
>currently hardcoded extensions), ctypes_load_library should do:
>import distutils.config
>so_ext = distutils.sysconfig.get_config_var('SO')
>to figure out the extension it should use to load shared libraries. This
>should make it reasonably easy for people to build shared libraries with
>distutils and use them with NumPy and ctypes.
>Comments appreciated. Someone checking something along these lines into SVN
>appreciated more. A solution that doesn't make me want to cry appreciated
>Thanks for reading.
>P.S. As it happens, the OOF2 guys have already created a SharedLibrary
>builder for distutils, but integrating this into numpy.distutils is probably
>Using Tomcat but need to do more? Need to support web services, security?
>Get stuff done quickly with pre-integrated technology to make your job easier
>Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
>Numpy-discussion mailing list
>Numpy-discussion at lists.sourceforge.net

More information about the Numpy-discussion mailing list