[IPython-user] ipython1: can't push class instance

Brian Granger ellisonbg.net@gmail....
Thu Dec 20 11:25:40 CST 2007


Unfortunately this is due to the details of how python pickles class
instances.  For a class to be pickleable, its definition needs to be
in a separate file/module/package that is importable from all
processes that unpickle/pickle it.  This, you should put your class
definition in one file:

# files foo.py

class Foo(object):
    pass

And then make sure that all processes can import that module:

* The ipython client
* The controller
* All of the engines

If you are running on a cluster, this could involve copying the file
to a bunch of machines.  I'll readily admit that this is highly
annoying, but it is sort of out of our control.  There are some hacks
you can do to get around this, but they don't always work and are
moderately hackish.

In the most recent version of IPython functions can be pushed and
pulled as long as they don't contain closures.  This is true even if
the function are defined interactively.  Because of this, I think it
is better to organize your code as follows:

1) Put all the low-level logic that does change often in classes and
distribute those files to all the machines.

2) Push and pull functions around for logic that is changing more often.

If you do this, you will be able to push/pull class instances around,
but if you want to change the class def at runtime, you will have to
re-distribute the files again.

Let us know if you are still having problems.

Brian

On Dec 20, 2007 7:45 AM, Vincent Schut <schut@sarvision.nl> wrote:
> Hi list,
>
> I'm trying to get started with ipython1, to investigate its parallel
> processing features.
> First of all a great kudos to the developers. Starting up and all is
> very simple, well documented, and works great.
>
> However, as far as I understand I should not only be able to pushAll
> simple values (like 2.34, "bla", etc) around to my engines, but also
> instances of my own classes. For our data processing, this would be
> crucial. However, whenever I try to pushAll an instance of even the
> simplest class possible, I get an error back.
>
> This is on gentoo linux, amd64, python 2.5.1, IPython and IPython1 from
> svn trunk as of this morning.
>
> Simple testcase:
>
> ### start source code ###
> #!/usr/bin/env python
> #coding=utf-8
>
> import ipython1.kernel.api as kernel
>
> class A(object):
>     pass
>
> if __name__=="__main__":
>
>     rc = kernel.RemoteController(('127.0.0.1',10105))
>     rc.pushAll(a=A())
>     print rc.executeAll("print a")
> ### end source code ###
>
> This fails for me with the following backtrace:
>
> Traceback (most recent call last):
>   File "/home/vincent/src/python/purschuim5/ipython1_test_001.py", line
> 26, in <module>
>     rc.pushAll(a=A())
>   File
> "/usr/lib64/python2.5/site-packages/ipython1-0.9alpha3-py2.5.egg/ipython1/kernel/multienginexmlrpc.py",
> line 631, in pushAll
>     return self.push('all', **ns)
>   File
> "/usr/lib64/python2.5/site-packages/ipython1-0.9alpha3-py2.5.egg/ipython1/kernel/multienginexmlrpc.py",
> line 621, in push
>     result = self._executeRemoteMethod(self._server.push,
> self._clientID, localBlock, targets, binPackage)
>   File
> "/usr/lib64/python2.5/site-packages/ipython1-0.9alpha3-py2.5.egg/ipython1/kernel/multienginexmlrpc.py",
> line 450, in _executeRemoteMethod
>     result = self._unpackageResult(rawResult)
>   File
> "/usr/lib64/python2.5/site-packages/ipython1-0.9alpha3-py2.5.egg/ipython1/kernel/multienginexmlrpc.py",
> line 459, in _unpackageResult
>     return self._returnOrRaise(result)
>   File
> "/usr/lib64/python2.5/site-packages/ipython1-0.9alpha3-py2.5.egg/ipython1/kernel/multienginexmlrpc.py",
> line 463, in _returnOrRaise
>     result.raiseException()
>   File "/usr/lib64/python2.5/site-packages/twisted/python/failure.py",
> line 259, in raiseException
>     raise self.type, self.value, self.tb
> AttributeError: 'module' object has no attribute 'A'
>
> I also tried a version where I, before the pushAll, had an 'from
> <filename> import A' executed on all engines, but it made no difference.
> I can push around numbers, strings, even a module like 'math' gets
> pushed away OK. Why not my class instance?
> Any hints on how to get this working?
>
> Cheers,
> Vincent Schut.
>
> _______________________________________________
> IPython-user mailing list
> IPython-user@scipy.org
> http://lists.ipython.scipy.org/mailman/listinfo/ipython-user
>


More information about the IPython-user mailing list