[SciPy-dev] Thoughts on weave improvements
eric at scipy.org
Tue Feb 12 11:06:30 CST 2002
> Eric and Pat, thanks for explaining the philosphy of weave and pycod, it
> cleared many things out for me. And I feel better now after knowing that
> there are ways to get weave/pycod efficient and really useful.
> On Mon, 11 Feb 2002, eric wrote:
> > I guess we'll have to poll you at the end of the summer to see if we've
> > weave... :) changed your mind. One of its major benefits (C callbacks)
> > require some cooperation with f2py so that Fortran wrappers check
> > the passed in object has a C representation to call instead of
> > calling back into Python. We'll try and make this as easy as possible,
> > of course, you'll have to sign on as a weave believer for the
> > work well.
> In fact, I was thinking the same thing for f2py, that is, if f2py'd
> extension function gets a call-back argument that is itself a f2py'd
> function (an therefore has a pointer to a real C/Fortran function), then
> it is called directly and not by the f2py call-back mechanism.
> The implementation of this feature is quite easy in f2py. What kept me
> implementing all that are the results of the timing measurments of the
> current f2py implementation. Namely, it turns out that the f2py call-back
> mechanism is already quite efficent (the time spent in doing this magic
> was only about 0.001 milli-seconds per call, error factor is 10) and most
> of the time (say 99%) is spent in Python (the test Python function
> contained only few scalar multiplications, usage of math.sin, and no
> array operations and loops).
> So, it is really crucial to get Python functions efficient and (may
> be) later optimize the interfaces (with a possibility of introducing new
> bugs;). Seems like weave/pycod will provide a solution here.
Just to make sure we're talking about the same thing, consider that we have
foo = weave.accelerate(foo)
foo is now some strange object. It is not a function call, but it is
callable so that the casual users will not know the difference. Inside this
foo object, will be multiple representations of the foo function. In the
simplest case, there will be 3. We'll call them "pure python," "extension
wrapper," and "pure C." The pure python version will basically be a pointer
to the Python function above. The other two will be in C code, and will
depend upon the type signatures of a and b. We'll say they are both int
values for simplicity -- this can be determined dynamically or specified by
the user. Here is what the C functions will look like -- the first
generated by pycod's machinery and the second by the current weave's
int pure_c_foo(int a, int b)
PyObject* foo_wrap(PyObject* self, PyObject* args)
// this is pseudo code
int a = py_to_int(targs);
int b = py_to_int(targs);
int c_result = pure_c_foo(a,b);
PyObject* py_result = int_to_py(c_result);
Generally, f2py calls back into Python into foo. It could be converted to
call foo_wrap directly with some savings, but the big win is having Fortran
functions call pure_c_foo directly so that Python is taken out of the loop
completely. f2py will need to ask accelerated objects if the have a pure C
implementation. If they do, it'll use this pointer, and everything happens
as fast as computer-ly possible. If they do not, they either ask the object
to compile a new version with the appropriate types, or call the pure python
version using the current callback scheme.
> You may want to look at fortranobject.c in f2py. Basically,
> 1) it implements a new type, FortranObject, and the objects of this type
> hold C pointers to real C/Fortran functions. These functions are called
> from f2py generated extension functions by using the corresponding
> 2) Also, the FortranObject type implements few methods:
> __call__ - so that in Python objects of type FortranObject behave like
> normal (well, almost) Python functions,
> __get/setattr__ - to return __doc__ strings and also to access Fortran
> common blocks or module data, the items of are also returned as
> The idea is simple but the implementation may look complex, that is
> because of Fortran specific stuff (just ignore it).
Will do. thanks for the pointer.
> I guess that the bottom line is that f2py and weave/pycod have much in
> common regarding the calls to real C/Fortran functions in run-time and I
> would hope that these common parts could be shared in some or another way.
> Do we need to introduce some independent (from f2py, weave/pycod,
> etc.) specification for objects holding C/Fortran objects and that
> implement some common tasks that are useful for both f2py and weave?
> Or, may be weave/pycod implement their own, say, CPointerObject, and I
> can easily implement hooks for such objects for f2py (as all that f2py
> needs to use are the C pointers to C/Fortran functions or data).
Not sure yet. If we find overlap, we should defintely get rid of it -- much
like we did in scipy_distutils. United code bases mean more brains working
on the same thing and smaller overall code bases. Both good things. Let us
get a bit farther on merging weave and pycod before we address this though.
I'm still trying to get a handle on how the caching mechanism should work
for this, and Pat is trying to get Python loop conversion to C working.
One other thought. weave already handles SWIG pointers (at least wxPython
versions of the things). The representation is pretty simple (a Python
string) and includes type information. Our first cut at representing
pointers will probably use SWIG's representation since it has served Dave B.
so well, and he's already written the code to handle them. :-) I think they
will prove to be sufficient.
I'm glad to see you've got ideas about all this also. weave and f2py
working in concert will be a big win (I hope...).
More information about the Scipy-dev