[IPython-User] Parallel direct view pull and bandwidth

[RICHARD Georges-Emmanuel]

Hi Minrk,

I got some doubt on my json setup, so I did few profiling:

data=[0.0 for i in range(62500)]
T=time.time();tmp=simplejson.loads(simplejson.dumps(data));print 
time.time() - T
0.087 s
T=time.time();tmp=json.loads(json.dumps(data));print time.time() - T
1.29 s
T=time.time();tmp=jsonlib2.read(jsonlib2.write(data));print time.time() - T
0.091 s
T=time.time();tmp=zmq.utils.jsonapi.loads(zmq.utils.jsonapi.dumps(data));print 
time.time() - T
0.09 s
T=time.time();tmp=ns.unserialize(ns.serialize(data));print time.time() - T
0.01 s  (IPython newserialise based on cPickle)

So I conclued the setup use jsonlib2 or cPickle to do the job, 
un/serialize doesn't seems to impact previous bench.

> in case of machine"A" as controller and as 2 ipengines
> pull FOO from machine"A" 2 engines 3.4   seconds        (2*500kB/3.4 
> => 294kB/s) on a local machine
> pull FOO from machine"A" 1 engine 2.7     seconds        (500kB/2.7 => 
> 185kB/s)

so I will continue to dig in on this point, although I could save the 
variables on each engines to the shared filesystem and load it back 
directly from the ipcontroller Client sides, but looks disgraceful.

else I continued on local machine to do some other bench:

  with 2 engines
T=time.time();dview.execute('FOO=0',block=False);print time.time() - T
     time is in range of 1 to1.5 ms    (which is great, compared to some 
old bench on a setup with ipython 0.8, which was around 40ms)
T=time.time();dview.execute('FOO=0',block=True);print time.time() - T
     time is in range of 5 to7 ms        ( still great)

with 2 engines
T=time.time();dview.pull('FOO').r;print time.time() - T
     time is in range of 4 to 6 ms    (which is great, compared to some 
old bench on a setup with ipython 0.8, which was around 80ms)
with 1 engine only
T=time.time();dview.pull('FOO',[0]).r;print time.time() - T
     time is in range of 3 to 5 ms    (which is great, compared to some 
old bench on a setup with ipython 0.8, which was around 80ms)

I finally found the slide of Fernando for Austin conf. and the page 15 
(Phenomenal task latency) confirm those last bench. But the throughput 
in page 16, I'm far.

I plan later to increase the number of engines to 32 with 17 computers 
(1 dedicated for the ipcontroller), I'll keep you updated of results.


Cheers.
             Joe
-- 

RICHARD Georges-Emmanuel
Electronic and Computer Engineer
perspective.electronic@gmail.com