[IPython-User] [kernel.client] map issues

Roman Yurchak rth.yurchak@gmail....
Fri Mar 25 16:12:15 CDT 2011


Hello,

I'm currently trying to make a basic parallel map using IPython's
kernel.client and I run into two issues.

-- Issue 1 --

 When the function that is to be called  has multiple
arguments and is mapped using MultiEngineClient for instance as:
      mec.map(lambda a: f(a,**c_args),np.arange(100))          (1)
where c_args = dict(b=10,c=3). This is equivalent to:
      mec.map(lambda a: f(a,10,3),np.arange(100))              (2)

The point is the formulation (1) works in the __main__ but fails when
called from a class method, while formulation (2) works all the time.
Just in case I have pushed c_args with MultiEngineClient

I get the following error:
    ValueError: Sorry, cannot pickle code objects with closures

I can not use formulation (2) as in real application c_args depends on
the current object properties.

You can find a minimal working example here:
http://perso.crans.org/~yurchak/ipython/mwe.py
 as well as the traceback:
http://perso.crans.org/~yurchak/ipython/mwe.log

--Issue 2--

As for functions that I managed to map with MultiEngineClient the
execution was quite slow, so I made a short benchmark using this file on
3 processes:
http://perso.crans.org/~yurchak/ipython/benchmark.py
and the ouput shows that for me the normal map is always faster than
mec.map while TaskClient's map just hungs using all available CPU power.
see http://perso.crans.org/~yurchak/ipython/ipython_latency.png

Additional logs can be found here:
http://perso.crans.org/~yurchak/ipython/ipcontroller.log
http://perso.crans.org/~yurchak/ipython/log/

Well I'm new to Ipython parallel processing capabilities, so am I
missing something here ? Or are there some issues with the software
versions I'm using ?

I use Gentoo with
Python 2.7.1 (r271:86832, Feb 28 2011, 13:20:38)
IPython 0.10.1 USE flags: (examples gnuplot readline -doc -emacs -smp
-test -wxwidgets)
dev-python/twisted   10.2.0
net-zope/zope-interface 3.6.1

Thank you in advance for your help,
best regards,
-- 
rth





More information about the IPython-User mailing list