[IPython-user] Re: IPython and psyco

David J. C. Beach beach at verinet.com
Tue May 18 09:57:17 CDT 2004

On Tue, 2004-05-18 at 01:20, Fernando Perez wrote:
> Where should such calls go?  And is there really any measurable impact?  If 
> this is something with a significant impact I could mention it in the 
> documentation.  Can you comment a bit on your experience with it?  I've never 
> used psyco myself, so I'm quite curious.

Personally, I have no need to accelerate IPython itself, so I've never
tried to make psyco optimize this code.  (To me, this doesn't make
sense, anyway... IPython is an interactive shell, and it only needs to
be "fast enough" for humans, which is alread is....)

I've never used psyco in a piece of code I've deployed.  (In other
words, I'm not sure that I trust it that much yet.)  However, psyco is
very nice for some internal things.  The best place to get started with
psyco is to read the documentation (repeatedly).  The way that it
optimizes Python code is absolutely fascinating.

[I am _not_ an authority on psyco; but here's an overview just the
same.]  Psyco operates by detecting the types in a particular python
function at run time, and then tries to write a specialized (assembly)
version of the function as the program runs.  Each time a new set of
types are found (for the same variables in the same function), psyco
writes a new, specialized version of that function for those types. 
This is the reason that pysco is called a specializing compiler.  As an
added trick (and important speedup), psyco tries to "virtualize" as many
python objects as possible.  Pysco has special knowledge of certain
builtin types such as int, string, tuple, array, and list.  If one of
these types is used completely within a function, and the instance never
leaves the function body (is exposed as a return value or side-effect),
then psyco can avoid the creation of the object altogether by in-lining
the object in assembly.

For more (much higher quality) information on this, go to the psyco
website and read the manual.  After a week goes by, read it again. :) 
There's also a nice graphical presentation on how psyco works which you
can download from the website [pygame required].

I've personally seen psyco accelerate python anywhere from 2x to about
100x, with an average speedup of around 2-3x.  In practice, you will
only get the 100x speedups if you write Python code with an
understanding of how psyco is able to optimize it, and even then, there
are only certain types of code which psyco can speed up this much. 
Generally speaking, pysco does it's very best at optimizing code which
uses integers, string, tuples, and arrays (i.e. the array module) very
very well.  Lists get some speedup, but are less likely to go as fast in
my experience.  Floats also get a speedup, but pysco still makes calls
into the Python API for these, so they don't work as fast as they could.

When invoking psyco, the user has several options.  The easiest is to
use "import psyco; psyco.full()", but this technique will not work
within IPython, because IPython counts on things like sys._getframe(),
locals() and globals(), which psyco cannot provide from within an
accelerated function (this information gets optimized away).  For 99.5%
of python programs, this is not a problem, but for IPython, it is. 
[There may be a way to tell Psyco specifically _not_ to optimize certain
functions/methods, and perhaps this logic could be added to the IPython

For larger programs, a better way to invoke psyco is to use the
aforementioned "psyco.bind()" call.  This instructs psyco to bind itself
only to specific functions/methods in the program.  In a large program,
usually only 20% or less is worth optimizing, so psyco.bind() becomes
the better way to do things.  Using this method will instruct psyco to
only accelerate specific methods, and this technique should be
compatible with IPython.


David J. C. Beach
<beach at verinet.com>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://projects.scipy.org/pipermail/ipython-user/attachments/20040518/dd085928/attachment-0002.bin 

More information about the IPython-user mailing list