[IPython-User] Memory usage of ipcontroller

[Brian Granger]

Giovanni,

I think that you can use TaskClient.clear() to remote the old
references.  Can you give that a shot?

Brian

On Mon, Sep 13, 2010 at 2:05 AM, Giovanni Luca Ciampaglia
<giovanni.luca.ciampaglia@usi.ch> wrote:
>  Hello everybody,
> I am running a small cluster with ipcontroller and ipengine and I am
> using the TaskClient interface to distribute simulations across 4
> workstations in the following way:
>>         try:
>>             results = self.tc.map(_simulate_engine,workload)
>>             return reduce(list.__add__,results)
>>         except CompositeError,e:
>>             getLogger('simulator.manager').error('Got error from
>> engines.',
>>                     exc_info=1)
>>             print e.print_tracebacks()
>
> It works great, though I noticed that when simulation results are large,
> ipcontroller keeps a lot of memory allocated after the simulation is
> done. Attached is a proof of concept using MultiEngineClient. If I run
> it with
>> ./test_ipcontroller.py ../furl/client-mec.furl 30000000
> ipcontroller keeps approximately 24% memory on a machine with 4GB of RAM
> after the script is done. I am using IPython 0.10 with Python 2.6.5 and
> NumPy 1.4.1
>
> ***********
>
> #!/usr/bin/env python
> import warnings
> with warnings.catch_warnings():
>     warnings.simplefilter('ignore', category=DeprecationWarning)
>     from IPython.kernel.client import MultiEngineClient
> import numpy as np
>
> def fat_return(N):
>     import numpy as np
>     return np.random.rand(N)
>
> if __name__ == '__main__':
>     import sys
>     if len(sys.argv) < 3:
>         print >> sys.stderr, 'usage: %s mec-furl-file N' % sys.argv[0]
>         sys.exit(-1)
>     with warnings.catch_warnings():
>         warnings.simplefilter('ignore', category=DeprecationWarning)
>         mec = MultiEngineClient(sys.argv[1])
>     N = int(sys.argv[2])
>     result = mec.map(fat_return,(N,)*len(mec.get_ids()))
>     print np.mean(result)
>
> --
> Giovanni L. Ciampaglia
> PhD Student
> University of Lugano, MACS Lab
>
> _______________________________________________
> IPython-User mailing list
> IPython-User@scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-user
>



-- 
Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo
bgranger@calpoly.edu
ellisonbg@gmail.com