[SciPy-dev] Thoughts on weave improvements

[Prabhu Ramachandran]

Hi,

>>>>> "PM" == Pat Miller <pnmiller at pacbell.net> writes:

    PM> Prabhu Ramachandran wrote:

    >> All this sounds very promising.  I just hope it becomes part of
    >> weave

    PM> Guess what.... It works (some restrictions and caveats of
    PM> course :-)

Wow!

    PM> # Get one of those nice weave module builders M =
    PM> weave.ext_tools.ext_module('foobar')

    PM> # Define some handy python function def f(x): "doc strings get
    PM> propagated" return x*x

    PM> # Here's where we define the accelerated function # Note that
    PM> you need to give the input signature # of the function
    PM> E.add_function( weaver.Python2CXX( f, [FloatType] ) )

I guess you need to do E = W somewhere.  Anyway, I get the idea.

[snip]

    PM> # OK... It isn't everything, but it can make a lot of simple
    PM> stuff run # about 2 to 5 times faster (real speed mostly eaten
    PM> up with function call # overhead for these really small
    PM> functions)

Yes, this is very cool!

    PM> Here are things that don't work...

[snip]

    PM> Still, not too shabby for a day and a half development time.

Thats real impressive indeed!

    PM> I'll be patching up holes and writing some more extensive
    PM> tests for this.  There are three main enhancements that will
    PM> get this kicking butt...

[snip]

    PM> 4) Build an interface to the struct module so that you can
    PM> build and use real C structs

Does this mean that you can actually accelerate a full fledged Python
class?

    PM> My goal here, in case it isn't obvious, is to allow users to
    PM> write modules completely in Python, debug and develop in a
    PM> great language, and then recover the speed through
    PM> compilation.  This has a side benefit in that Jython and
    PM> PalmPilot-thon and Embedded-processor-thon can all use the
    PM> Python variant of your code and if you run on a "real" machine
    PM> you get the speed.

Yes, it sure would be a *very* nice goal.  One thing I cant but help
noticing is that it looks like this is all heading towards something
like Dylan.  Its like Python + type info + magical parser that parses
functions and produces c/c++ extensions.

I ask a naive question, what is the feasibility of actually doing
something like Dylan?  OK forget truly generic functions.  I'm just
talking of optional typing to speed things up.  I guess psyco, weave,
pyCOD etc. all put together and taken to their logical end will give
us Python + optional typing + heaven? :D Of course I could also be
*completely* mistaken.  Looking at Eric's subsequent post it looks
like even optional typing can be magically handled.  This is looking
amazing.

How far can we go?  What would be possible and what is simply out of
question??  If only a small subset of Python features are used by high
perf. Python code, what subset are we talking about??  Wrapping a
function is fine.  What about classes?  Would that be possible.  I
know maybe I'm wishing for too much but I think its important to
atleast understand what is possible.

    PM> Hope this is useful stuff.  If you want the code, drop me a
    PM> line and I'll pass it along.  I'm hoping that Eric will let me
    PM> integrate this directly with weave.  I think it is good enough

I'm sure Eric will be very happy to let you integrate this with weave
as will a lot of other users!

    PM> to use out of the box (since it doesn't barf at every
    PM> opportunity), but it will take some shaking out to get it to
    PM> really work and this can only happen with users!  Let me know
    PM> who you are...

Well, count me in but I am busy with too many things these days but I
will try to experiment with it when I have the time.

prabhu