[SciPy-user] Parallel OO programming in Python.

Brian Granger ellisonbg.net at gmail.com
Mon Oct 30 16:00:22 CST 2006


It looks like you are using an older version of zope.interface.  I
would grab this version and try it.

http://www.zope.org/Products/ZopeInterface/3.1.0c1/ZopeInterface-3.1.0c1.tgz

also, I would use a recent (I alway use the svn trunk) version of
twisted.  Let me know if that helps.

> I am playing with it right now!  I am planning to run it soon on multiple
> machines with multiple CPUs. Will I have to install Ipython1 on every node
> or Will the controller host spawn the processes across the net? In the
> README you seem to be starting the engines locally on each host....

Yes, you will need to install everything on each host and start the
engines on each machine.  If you have mpi installed, you can start the
engines using mpirun and this will handle it.  Let us know how it
goes.

Brian

> but before I get Ahead of myself , I am getting the following error when I
> try to start an engine on my localhost:
> fccoelho at sombra ~/Downloads/Twisted $ ipengine &
> [2] 32018
> fccoelho at sombra ~/Downloads/Twisted $ 2006/10/30 16:44 -0200 [-] Log opened.
> 2006/10/30 16:44 -0200 [-] Starting factory
> <ipython1.kernel.enginevanilla.VanillaEngineClientFactoryFromEngineService
> object at 0xb6706eac>
> 2006/10/30 16:44 -0200 [VanillaEngineServerProtocol,0, 127.0.0.1] Unhandled
> Error
>         Traceback (most recent call last):
>           File
> "/usr/lib/python2.4/site-packages/twisted/python/log.py",
> line 48, in callWithLogger
>             return callWithContext({"system": lp}, func, *args, **kw)
>           File
> "/usr/lib/python2.4/site-packages/twisted/python/log.py",
> line 33, in callWithContext
>             return context.call({ILogContext: newCtx}, func, *args, **kw)
>           File
> "/usr/lib/python2.4/site-packages/twisted/python/context.py",
> line 59, in callWithContext
>             return self.currentContext().callWithContext(ctx, func, *args,
> **kw)
>           File
> "/usr/lib/python2.4/site-packages/twisted/python/context.py",
> line 37, in callWithContext
>             return func(*args,**kw)
>         --- <exception caught here> ---
>           File
> "/usr/lib/python2.4/site-packages/twisted/internet/selectreactor.py",
> line 139, in _doReadOrWrite
>             why = getattr(selectable, method)()
>           File
> "/usr/lib/python2.4/site-packages/twisted/internet/tcp.py",
> line 362, in doRead
>             return self.protocol.dataReceived(data)
>           File
> "/usr/lib/python2.4/site-packages/twisted/protocols/basic.py",
> line 99, in dataReceived
>             self.doData()
>           File
> "/usr/lib/python2.4/site-packages/twisted/protocols/basic.py",
> line 62, in doData
>             self.stringReceived(self.__buffer)
>           File
> "/usr/lib/python2.4/site-packages/ipython1/kernel/enginevanilla.py",
> line 557, in stringReceived
>             self.nextHandler(msg)
>           File
> "/usr/lib/python2.4/site-packages/ipython1/kernel/enginevanilla.py",
> line 573, in dispatch
>             f(args)
>           File
> "/usr/lib/python2.4/site-packages/ipython1/kernel/enginevanilla.py",
> line 706, in handle_REGISTER
>             qe = engineservice.QueuedEngine(self, keepUpToDate=True)
>           File
> "/usr/lib/python2.4/site-packages/ipython1/kernel/engineservice.py",
> line 342, in __init__
>             self.registerMethods ()
>           File
> "/usr/lib/python2.4/site-packages/ipython1/kernel/engineservice.py",
> line 345, in registerMethods
>             zi.alsoProvides(self, *self.engine.__provides__)
>         exceptions.AttributeError : 'module' object has no attribute
> 'alsoProvides'
>
> 2006/10/30 16:44 -0200 [VanillaEngineServerProtocol,0,127.0.0.1]
> unregistered engine None
> 2006/10/30 16:44 -0200 [VanillaEngineClientProtocol,client]
> Stopping factory <
> ipython1.kernel.enginevanilla.VanillaEngineClientFactoryFromEngineService
> object at 0xb6706eac>
> 2006/10/30 16:44 -0200 [VanillaEngineServerProtocol,0,127.0.0.1] Unhandled
> Error
>         Traceback (most recent call last):
>           File "/usr/bin/ipcontroller", line 51, in main
>             reactor.run()
>           File
> "/usr/lib/python2.4/site-packages/twisted/internet/posixbase.py",
> line 218, in run
>             self.mainLoop()
>           File
> "/usr/lib/python2.4/site-packages/twisted/internet/posixbase.py",
> line 229, in mainLoop
>             self.doIteration(t)
>           File
> "/usr/lib/python2.4/site-packages/twisted/internet/selectreactor.py",
> line 133, in doSelect
>             _logrun(selectable, _drdw, selectable, method, dict)
>         --- <exception caught here> ---
>           File
> "/usr/lib/python2.4/site-packages/twisted/python/log.py",
> line 48, in callWithLogger
>             return callWithContext({"system": lp}, func, *args, **kw)
>           File
> "/usr/lib/python2.4/site-packages/twisted/python/log.py",
> line 33, in callWithContext
>             return context.call ({ILogContext: newCtx}, func, *args, **kw)
>           File
> "/usr/lib/python2.4/site-packages/twisted/python/context.py",
> line 59, in callWithContext
>             return self.currentContext().callWithContext(ctx, func, *args,
> **kw)
>           File
> "/usr/lib/python2.4/site-packages/twisted/python/context.py",
> line 37, in callWithContext
>             return func(*args,**kw)
>           File
> "/usr/lib/python2.4/site-packages/twisted/internet/selectreactor.py",
> line 149, in _doReadOrWrite
>             self._disconnectSelectable(selectable, why,
> method=="doRead")
>           File
> "/usr/lib/python2.4/site-packages/twisted/internet/posixbase.py",
> line 256, in _disconnectSelectable
>             selectable.connectionLost(failure.Failure(why))
>           File
> "/usr/lib/python2.4/site-packages/twisted/internet/tcp.py",
> line 416, in connectionLost
>             protocol.connectionLost(reason)
>           File
> "/usr/lib/python2.4/site-packages/ipython1/kernel/enginevanilla.py",
> line 538, in connectionLost
>             self.factory.unregisterEngine(self.id)
>           File
> "/usr/lib/python2.4/site-packages/ipython1/kernel/enginevanilla.py",
> line 1023, in unregisterEngine
>             return self.service.unregisterEngine(id)
>           File
> "/usr/lib/python2.4/site-packages/ipython1/kernel/controllerservice.py",
> line 300, in unregisterEngine
>             del self.engines[id]
>         exceptions.KeyError :
>
>
>
> On 10/30/06, Brian Granger <ellisonbg.net at gmail.com> wrote:
> >
> > > Can Ipython1 take advantage of multiple core cpus or SMP machines in the
> > > same way it does for a networked cluster? It would be nice to have that
> > > since quad-core cpu are arriving on the market and SMP machines are
> pretty
> > > common already.
> >
> > This was one of the major usage cases we have had in mind.  So yes.
> > Some details though:
> >
> > In IPython1 computations are done by a separate process that we are
> > calling the IPython engine.  Multiple IPython engines can be run on
> > multiple systems (distributed memory), a single (multi CPU and/or
> > multi-core) system or a combination of the two.  I have run two
> > IPython engines on my dual core MacBook and seen 2x speedup with
> > simple python scripts.  I wasn't using MPI, but with MPi you should
> > see a similar behavior as long as your algorithm scales linearly to 2
> > instances processes.
> >
> > I can't wait to use IPython1 with 4/8/16 cores on my workstation :)
> >
> > Please let us know if you have questions or problems.
> >
> > Brian
> >
> > On 10/30/06, Flavio Coelho <fccoelho at gmail.com> wrote:
> > > Hi Brian,
> > >
> > > Can Ipython1 take advantage of multiple core cpus or SMP machines in the
> > > same way it does for a networked cluster? It would be nice to have that
> > > since quad-core cpu are arriving on the market and SMP machines are
> pretty
> > > common already.
> > >
> > > Thanks!
> > >
> > > Flávio
> > >
> > >
> > > On 10/30/06, Brian Granger <ellisonbg.net at gmail.com> wrote:
> > > > > I also looked at the entry on Ipython1 and it looks really
> promising.
> > > > > Did I understand correctly that Ipython1 is able to run parallel
> python
> > > > > code, regardless of which "MPI for python" package one eventually
> uses?
> > > >
> > > > In principle yes.  The IPython engines have a configuration option
> > > > that you use to import the mpi bindings of your choice.  But, to work
> > > > properly, the import statement must cause MPI_Init() to be called.  If
> > > > the bindings you are using don't do that, it should b easy to write a
> > > > wrapper module that imports and also calls MPI_Init().  Also, some of
> > > > this depends on what MPI implementation you are using.  The more
> > > > moderns ones (like openmpi) are really nice and pretty forgiving.  The
> > > > older ones, though, sometimes require that Python itself (not your
> > > > script) call MPI_Init().  This is also very easy to work around.  The
> > > > bottom line is that we can help you get it to work with whatever mpi
> > > > binding and implementation you want.
> > > >
> > > > Withh that said, I highly do highly recommend mpi4py and openmpi - we
> > > > use these regularly with ipython and it all works well.
> > > >
> > > > Brian
> > > >
> > > >
> > > > > Giovanni
> > > > >
> > > > > Flavio Coelho wrote:
> > > > > > Thank you for this post and many thanks to Fernando Perez, for the
> > > > > > awesome tool that Ipython1 is becomming!!! I am beginning to use
> it
> > > > > > today!!
> > > > > >
> > > > > > Flávio
> > > > > >
> > > > > > On 10/29/06, * Brian Granger* < ellisonbg.net at gmail.com
> > > > > > <mailto:ellisonbg.net at gmail.com>> wrote:
> > > > > >
> > > > > >     > For object oriented parallel programming in Python on a
> single
> > > > > >     machine I can use techniques such as Bulk Synchronous
> Parallelism
> > > > > >     (BSP) or Message Passing Interface (MPI). There are mentioned
> some
> > > > > >     modules and packages for Python on
> > > > > >     > http://www.scipy.org/Topical_Software
> > > > > >     > Article: Parallel and distributed programming.
> > > > > >     > After reading prerequisites and limitations I thing that the
> > > > > >     following ones could be good for me:
> > > > > >     > PyMPI
> > > > > >     > Pypar
> > > > > >     > MPI for Python
> > > > > >
> > > > > >     I think mpi4py is the best option.  It is a very nice
> > > implementation
> > > > > >     and the developer is working really hard to make it a great
> > > package.
> > > > > >     Also, I don't think pympi and pypar are being developed
> actively
> > > > > >     anymore.
> > > > > >
> > > > > >     Another thing you might check out is IPython's parallel
> computing
> > > > > >     facilities that let you use mpi4py (and other mpi bindings)
> > > > > >     interactively:
> > > > > >
> > > > > >
> http://ipython.scipy.org/moin/Parallel_Computing
> > > > > >
> > > > > >     I have never used BPS, but am familiar with the model.  Can
> > > probably
> > > > > >     get it working interactively with IPython as well.
> > > > > >
> > > > > >     Cheers,
> > > > > >
> > > > > >     Brian
> > > > > >
> > > > > >     > But I can't distinguish which one brings fewer limitations,
> is
> > > > > >     efficient in application and will develop in future.
> > > > > >     > My favorite is BSP, but I can't find a package for present
> SciPy
> > > > > >     and NumPy.
> > > > > >     >
> > > > > >     > Could anybody give me a recommendation, which way I should
> go.
> > > > > >     >
> > > > > >     > Thanks.
> > > > > >     > Fie Pye
> > > > > >     >
> > > > > >     >
> > > > > >     >
> > > > > >     >
> > > > > >     > ---------------------------------------
> > > > > >     > Máme další spot P.Čtvrtníčka. Podívejte se www.neuservis.cz
> > > > > >     < http://www.neuservis.cz>
> > > > > >     >
> > > > > >     >
> > > > > >     >
> _______________________________________________
> > > > > >     > SciPy-user mailing list
> > > > > >     > SciPy-user at scipy.org <mailto:SciPy-user at scipy.org>
> > > > > >     >
> > > http://projects.scipy.org/mailman/listinfo/scipy-user
> > > > > >
> > > <http://projects.scipy.org/mailman/listinfo/scipy-user
> >
> > > > > >     >
> > > > > >
> _______________________________________________
> > > > > >     SciPy-user mailing list
> > > > > >     SciPy-user at scipy.org <mailto: SciPy-user at scipy.org>
> > > > > >
> > > http://projects.scipy.org/mailman/listinfo/scipy-user
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Flávio Codeço Coelho
> > > > > > registered Linux user # 386432
> > > > > > ---------------------------
> > > > > > "Laws are like sausages. It's better not to see them being made."
> > > > > > Otto von Bismark
> > > > > >
> > >
> ------------------------------------------------------------------------
> > > > > >
> > > > > > _______________________________________________
> > > > > > SciPy-user mailing list
> > > > > > SciPy-user at scipy.org
> > > > > >
> http://projects.scipy.org/mailman/listinfo/scipy-user
> > > > > >
> > > > > _______________________________________________
> > > > > SciPy-user mailing list
> > > > > SciPy-user at scipy.org
> > > > >
> http://projects.scipy.org/mailman/listinfo/scipy-user
> > > > >
> > > > _______________________________________________
> > > > SciPy-user mailing list
> > > > SciPy-user at scipy.org
> > > > http://projects.scipy.org/mailman/listinfo/scipy-user
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > Flávio Codeço Coelho
> > > registered Linux user # 386432
> > > ---------------------------
> > > "Laws are like sausages. It's better not to see them being made."
> > > Otto von Bismark
> > > _______________________________________________
> > > SciPy-user mailing list
> > > SciPy-user at scipy.org
> > > http://projects.scipy.org/mailman/listinfo/scipy-user
> > >
> > >
> > >
> > _______________________________________________
> > SciPy-user mailing list
> > SciPy-user at scipy.org
> > http://projects.scipy.org/mailman/listinfo/scipy-user
> >
>
>
>
> --
> Flávio Codeço Coelho
> registered Linux user # 386432
> ---------------------------
> "Laws are like sausages. It's better not to see them being made."
> Otto von Bismark
> _______________________________________________
> SciPy-user mailing list
> SciPy-user at scipy.org
> http://projects.scipy.org/mailman/listinfo/scipy-user
>
>
>


More information about the SciPy-user mailing list