[IPython-User] [kernel.client] map issues
Fri Mar 25 16:12:15 CDT 2011
I'm currently trying to make a basic parallel map using IPython's
kernel.client and I run into two issues.
-- Issue 1 --
When the function that is to be called has multiple
arguments and is mapped using MultiEngineClient for instance as:
mec.map(lambda a: f(a,**c_args),np.arange(100)) (1)
where c_args = dict(b=10,c=3). This is equivalent to:
mec.map(lambda a: f(a,10,3),np.arange(100)) (2)
The point is the formulation (1) works in the __main__ but fails when
called from a class method, while formulation (2) works all the time.
Just in case I have pushed c_args with MultiEngineClient
I get the following error:
ValueError: Sorry, cannot pickle code objects with closures
I can not use formulation (2) as in real application c_args depends on
the current object properties.
You can find a minimal working example here:
as well as the traceback:
As for functions that I managed to map with MultiEngineClient the
execution was quite slow, so I made a short benchmark using this file on
and the ouput shows that for me the normal map is always faster than
mec.map while TaskClient's map just hungs using all available CPU power.
Additional logs can be found here:
Well I'm new to Ipython parallel processing capabilities, so am I
missing something here ? Or are there some issues with the software
versions I'm using ?
I use Gentoo with
Python 2.7.1 (r271:86832, Feb 28 2011, 13:20:38)
IPython 0.10.1 USE flags: (examples gnuplot readline -doc -emacs -smp
Thank you in advance for your help,
More information about the IPython-User