[IPython-User] Memory usage of ipcontroller

[Giovanni Luca Ciampaglia]

  Hello everybody,
I am running a small cluster with ipcontroller and ipengine and I am 
using the TaskClient interface to distribute simulations across 4 
workstations in the following way:
>         try:
>             results = self.tc.map(_simulate_engine,workload)
>             return reduce(list.__add__,results)
>         except CompositeError,e:
>             getLogger('simulator.manager').error('Got error from 
> engines.',
>                     exc_info=1)
>             print e.print_tracebacks()

It works great, though I noticed that when simulation results are large, 
ipcontroller keeps a lot of memory allocated after the simulation is 
done. Attached is a proof of concept using MultiEngineClient. If I run 
it with
> ./test_ipcontroller.py ../furl/client-mec.furl 30000000
ipcontroller keeps approximately 24% memory on a machine with 4GB of RAM 
after the script is done. I am using IPython 0.10 with Python 2.6.5 and 
NumPy 1.4.1

***********

#!/usr/bin/env python
import warnings
with warnings.catch_warnings():
     warnings.simplefilter('ignore', category=DeprecationWarning)
     from IPython.kernel.client import MultiEngineClient
import numpy as np

def fat_return(N):
     import numpy as np
     return np.random.rand(N)

if __name__ == '__main__':
     import sys
     if len(sys.argv) < 3:
         print >> sys.stderr, 'usage: %s mec-furl-file N' % sys.argv[0]
         sys.exit(-1)
     with warnings.catch_warnings():
         warnings.simplefilter('ignore', category=DeprecationWarning)
         mec = MultiEngineClient(sys.argv[1])
     N = int(sys.argv[2])
     result = mec.map(fat_return,(N,)*len(mec.get_ids()))
     print np.mean(result)

-- 
Giovanni L. Ciampaglia
PhD Student
University of Lugano, MACS Lab