[IPython-User] parallel question: how to really reset an engine

MinRK benjaminrk@gmail....
Fri Jun 22 15:31:04 CDT 2012


On Fri, Jun 22, 2012 at 12:38 PM, Toby Burnett <tburnett@uw.edu> wrote:

>  I’ve been using the parallel facility for some time, very productively. *
> ***
>
> My mode is to initialize all engines for a particular setup, then allocate
> cases to each one. A new run then involves a new setup, for which I’d like
> to  clear and reuse the engines.  I invoke  the parallel function clear,
> which clears the remote namespace OK, but the memory use does not change,
> and a new initialization adds more memory.
>

This sort of thing is incredibly difficult to diagnose without code (and
even pretty hard with code).  Every test I have done with simple numpy
arrays results in properly freed memory on the engines.


> ****
>
> Running ‘import gc; gc.collect()’  does not seem to help.****
>
> So, I really want to restart the python process without having to kill,
> and restart the engine.
>

Restart *is* killing and restarting (the restart in the notebook/qtconsole
is literally: kill, wait, restart).

One of the things that I have long wanted to add to the engines is
KernelManager capabilities, so that they can be interrupted/restarted like
the regular Kernels.

We have a bit of degeneracy in our terminology/implementation, because what
we call a KernelManager is really two very different things: a
KernelManager, and the base KernelClient.  There should only be one
KernelManager per kernel/engine, and then one KernelClient per frontend.
 This would allow remote signalling/interrupting/restarting, which we
currently don't have, and the same changes would enable the KernelManager
to be used by Engines.



****
>
> ** **
>
> --Toby Burnett****
>
> _______________________________________________
> IPython-User mailing list
> IPython-User@scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/ipython-user/attachments/20120622/b3c484e9/attachment-0001.html 


More information about the IPython-User mailing list