[IPython-User] Memory usage of ipcontroller

[Giovanni Luca Ciampaglia]

Brian,
TaskClient.clear and MultiEngineClient.clear_properties cleared half of 
the memory in the example below. Other clear_* methods (both in TC and 
MEC) didn't bring any other improvement. Thanks for the help!

G


On 13/09/2010 20:18, Brian Granger wrote:
> Giovanni,
>
> I think that you can use TaskClient.clear() to remote the old
> references.  Can you give that a shot?
>
> Brian
>
> On Mon, Sep 13, 2010 at 2:05 AM, Giovanni Luca Ciampaglia
> <giovanni.luca.ciampaglia@usi.ch>  wrote:
>>   Hello everybody,
>> I am running a small cluster with ipcontroller and ipengine and I am
>> using the TaskClient interface to distribute simulations across 4
>> workstations in the following way:
>>>          try:
>>>              results = self.tc.map(_simulate_engine,workload)
>>>              return reduce(list.__add__,results)
>>>          except CompositeError,e:
>>>              getLogger('simulator.manager').error('Got error from
>>> engines.',
>>>                      exc_info=1)
>>>              print e.print_tracebacks()
>>
>> It works great, though I noticed that when simulation results are large,
>> ipcontroller keeps a lot of memory allocated after the simulation is
>> done. Attached is a proof of concept using MultiEngineClient. If I run
>> it with
>>> ./test_ipcontroller.py ../furl/client-mec.furl 30000000
>> ipcontroller keeps approximately 24% memory on a machine with 4GB of RAM
>> after the script is done. I am using IPython 0.10 with Python 2.6.5 and
>> NumPy 1.4.1
>>
>> ***********
>>
>> #!/usr/bin/env python
>> import warnings
>> with warnings.catch_warnings():
>>      warnings.simplefilter('ignore', category=DeprecationWarning)
>>      from IPython.kernel.client import MultiEngineClient
>> import numpy as np
>>
>> def fat_return(N):
>>      import numpy as np
>>      return np.random.rand(N)
>>
>> if __name__ == '__main__':
>>      import sys
>>      if len(sys.argv)<  3:
>>          print>>  sys.stderr, 'usage: %s mec-furl-file N' % sys.argv[0]
>>          sys.exit(-1)
>>      with warnings.catch_warnings():
>>          warnings.simplefilter('ignore', category=DeprecationWarning)
>>          mec = MultiEngineClient(sys.argv[1])
>>      N = int(sys.argv[2])
>>      result = mec.map(fat_return,(N,)*len(mec.get_ids()))
>>      print np.mean(result)
>>
>> --
>> Giovanni L. Ciampaglia
>> PhD Student
>> University of Lugano, MACS Lab
>>
>> _______________________________________________
>> IPython-User mailing list
>> IPython-User@scipy.org
>> http://mail.scipy.org/mailman/listinfo/ipython-user
>>
>
>
>


-- 
Giovanni L. Ciampaglia
PhD Student
University of Lugano, MACS Lab