[IPython-user] ipython1: can't push class instance
Fri Dec 21 03:52:21 CST 2007
thanks for the clarification. Now that explains a lot, especially
because I really felt sure I remembered having this working some days
ago, and started to doubt my own memory capabilities... I knew it had to
be something simple, as I hadn't changed much and suddenly it wasn't
Having my classes in separate files is no problem, in fact it was just
for testing that I hadn't; the real-world processing stuff is already in
several classes inside a module, so that won't be a problem.
Also, all my engines share the same NFS mounted hard disk space,
including a ~/src/python that is automatically in the pythonpath. So I
do not foresee any problems related to importing. All engines and
controllers run as the same user with its home mounted from nfs.
Btw. this nfs setup is also there to avoid having to push around large
amounts of data (we process pretty large datasets, think at least
gigabytes, in the future maybe terabytes, so I'm trying to work out a
scalable solution...). I know of course it won't save me much on network
traffic (and might even increase it), however, it will save me much on
memory usage because the client/controller doesn't need to load the data
and push it away. I just tell the engines what part of the data to load,
and they load it from the nfs shared space, and save their results to it
(using locks of course).
I'll keep the list updated on any problems (or maybe significant
progress, success stories are always nice to read ;)).
Brian Granger wrote:
> Unfortunately this is due to the details of how python pickles class
> instances. For a class to be pickleable, its definition needs to be
> in a separate file/module/package that is importable from all
> processes that unpickle/pickle it. This, you should put your class
> definition in one file:
> # files foo.py
> class Foo(object):
> And then make sure that all processes can import that module:
> * The ipython client
> * The controller
> * All of the engines
> If you are running on a cluster, this could involve copying the file
> to a bunch of machines. I'll readily admit that this is highly
> annoying, but it is sort of out of our control. There are some hacks
> you can do to get around this, but they don't always work and are
> moderately hackish.
> In the most recent version of IPython functions can be pushed and
> pulled as long as they don't contain closures. This is true even if
> the function are defined interactively. Because of this, I think it
> is better to organize your code as follows:
> 1) Put all the low-level logic that does change often in classes and
> distribute those files to all the machines.
> 2) Push and pull functions around for logic that is changing more often.
> If you do this, you will be able to push/pull class instances around,
> but if you want to change the class def at runtime, you will have to
> re-distribute the files again.
> Let us know if you are still having problems.
> On Dec 20, 2007 7:45 AM, Vincent Schut <firstname.lastname@example.org> wrote:
>> Hi list,
>> I'm trying to get started with ipython1, to investigate its parallel
>> processing features.
>> First of all a great kudos to the developers. Starting up and all is
>> very simple, well documented, and works great.
>> However, as far as I understand I should not only be able to pushAll
>> simple values (like 2.34, "bla", etc) around to my engines, but also
>> instances of my own classes. For our data processing, this would be
>> crucial. However, whenever I try to pushAll an instance of even the
>> simplest class possible, I get an error back.
>> This is on gentoo linux, amd64, python 2.5.1, IPython and IPython1 from
>> svn trunk as of this morning.
>> Simple testcase:
>> ### start source code ###
>> #!/usr/bin/env python
>> import ipython1.kernel.api as kernel
>> class A(object):
>> if __name__=="__main__":
>> rc = kernel.RemoteController(('127.0.0.1',10105))
>> print rc.executeAll("print a")
>> ### end source code ###
>> This fails for me with the following backtrace:
>> Traceback (most recent call last):
>> File "/home/vincent/src/python/purschuim5/ipython1_test_001.py", line
>> 26, in <module>
>> line 631, in pushAll
>> return self.push('all', **ns)
>> line 621, in push
>> result = self._executeRemoteMethod(self._server.push,
>> self._clientID, localBlock, targets, binPackage)
>> line 450, in _executeRemoteMethod
>> result = self._unpackageResult(rawResult)
>> line 459, in _unpackageResult
>> return self._returnOrRaise(result)
>> line 463, in _returnOrRaise
>> File "/usr/lib64/python2.5/site-packages/twisted/python/failure.py",
>> line 259, in raiseException
>> raise self.type, self.value, self.tb
>> AttributeError: 'module' object has no attribute 'A'
>> I also tried a version where I, before the pushAll, had an 'from
>> <filename> import A' executed on all engines, but it made no difference.
>> I can push around numbers, strings, even a module like 'math' gets
>> pushed away OK. Why not my class instance?
>> Any hints on how to get this working?
>> Vincent Schut.
>> IPython-user mailing list
More information about the IPython-user