[IPython-user] ipython1: can't push class instance

Barry Wark barrywark@gmail....
Thu Dec 20 12:22:19 CST 2007


Would it be possible to ask ipython to push the files defining a class
to the controllers & engines? I suppose this would take a little
mucking with sys.path to add some private folder where these pushed
py/pyc files are stored, but is there any fundamental problem with
doing that? i realize that compiled extensions etc. might still not
work.

barry

On Dec 20, 2007 9:25 AM, Brian Granger <ellisonbg.net@gmail.com> wrote:
> Unfortunately this is due to the details of how python pickles class
> instances.  For a class to be pickleable, its definition needs to be
> in a separate file/module/package that is importable from all
> processes that unpickle/pickle it.  This, you should put your class
> definition in one file:
>
> # files foo.py
>
> class Foo(object):
>     pass
>
> And then make sure that all processes can import that module:
>
> * The ipython client
> * The controller
> * All of the engines
>
> If you are running on a cluster, this could involve copying the file
> to a bunch of machines.  I'll readily admit that this is highly
> annoying, but it is sort of out of our control.  There are some hacks
> you can do to get around this, but they don't always work and are
> moderately hackish.
>
> In the most recent version of IPython functions can be pushed and
> pulled as long as they don't contain closures.  This is true even if
> the function are defined interactively.  Because of this, I think it
> is better to organize your code as follows:
>
> 1) Put all the low-level logic that does change often in classes and
> distribute those files to all the machines.
>
> 2) Push and pull functions around for logic that is changing more often.
>
> If you do this, you will be able to push/pull class instances around,
> but if you want to change the class def at runtime, you will have to
> re-distribute the files again.
>
> Let us know if you are still having problems.
>
> Brian
>
>
> On Dec 20, 2007 7:45 AM, Vincent Schut <schut@sarvision.nl> wrote:
> > Hi list,
> >
> > I'm trying to get started with ipython1, to investigate its parallel
> > processing features.
> > First of all a great kudos to the developers. Starting up and all is
> > very simple, well documented, and works great.
> >
> > However, as far as I understand I should not only be able to pushAll
> > simple values (like 2.34, "bla", etc) around to my engines, but also
> > instances of my own classes. For our data processing, this would be
> > crucial. However, whenever I try to pushAll an instance of even the
> > simplest class possible, I get an error back.
> >
> > This is on gentoo linux, amd64, python 2.5.1, IPython and IPython1 from
> > svn trunk as of this morning.
> >
> > Simple testcase:
> >
> > ### start source code ###
> > #!/usr/bin/env python
> > #coding=utf-8
> >
> > import ipython1.kernel.api as kernel
> >
> > class A(object):
> >     pass
> >
> > if __name__=="__main__":
> >
> >     rc = kernel.RemoteController(('127.0.0.1',10105))
> >     rc.pushAll(a=A())
> >     print rc.executeAll("print a")
> > ### end source code ###
> >
> > This fails for me with the following backtrace:
> >
> > Traceback (most recent call last):
> >   File "/home/vincent/src/python/purschuim5/ipython1_test_001.py", line
> > 26, in <module>
> >     rc.pushAll(a=A())
> >   File
> > "/usr/lib64/python2.5/site-packages/ipython1-0.9alpha3-py2.5.egg/ipython1/kernel/multienginexmlrpc.py",
> > line 631, in pushAll
> >     return self.push('all', **ns)
> >   File
> > "/usr/lib64/python2.5/site-packages/ipython1-0.9alpha3-py2.5.egg/ipython1/kernel/multienginexmlrpc.py",
> > line 621, in push
> >     result = self._executeRemoteMethod(self._server.push,
> > self._clientID, localBlock, targets, binPackage)
> >   File
> > "/usr/lib64/python2.5/site-packages/ipython1-0.9alpha3-py2.5.egg/ipython1/kernel/multienginexmlrpc.py",
> > line 450, in _executeRemoteMethod
> >     result = self._unpackageResult(rawResult)
> >   File
> > "/usr/lib64/python2.5/site-packages/ipython1-0.9alpha3-py2.5.egg/ipython1/kernel/multienginexmlrpc.py",
> > line 459, in _unpackageResult
> >     return self._returnOrRaise(result)
> >   File
> > "/usr/lib64/python2.5/site-packages/ipython1-0.9alpha3-py2.5.egg/ipython1/kernel/multienginexmlrpc.py",
> > line 463, in _returnOrRaise
> >     result.raiseException()
> >   File "/usr/lib64/python2.5/site-packages/twisted/python/failure.py",
> > line 259, in raiseException
> >     raise self.type, self.value, self.tb
> > AttributeError: 'module' object has no attribute 'A'
> >
> > I also tried a version where I, before the pushAll, had an 'from
> > <filename> import A' executed on all engines, but it made no difference.
> > I can push around numbers, strings, even a module like 'math' gets
> > pushed away OK. Why not my class instance?
> > Any hints on how to get this working?
> >
> > Cheers,
> > Vincent Schut.
> >
> > _______________________________________________
> > IPython-user mailing list
> > IPython-user@scipy.org
> > http://lists.ipython.scipy.org/mailman/listinfo/ipython-user
> >
> _______________________________________________
> IPython-user mailing list
> IPython-user@scipy.org
> http://lists.ipython.scipy.org/mailman/listinfo/ipython-user
>


More information about the IPython-user mailing list