[IPython-User] parallel question: how to really reset an engine
Fri Jun 22 15:31:04 CDT 2012
On Fri, Jun 22, 2012 at 12:38 PM, Toby Burnett <email@example.com> wrote:
> I’ve been using the parallel facility for some time, very productively. *
> My mode is to initialize all engines for a particular setup, then allocate
> cases to each one. A new run then involves a new setup, for which I’d like
> to clear and reuse the engines. I invoke the parallel function clear,
> which clears the remote namespace OK, but the memory use does not change,
> and a new initialization adds more memory.
This sort of thing is incredibly difficult to diagnose without code (and
even pretty hard with code). Every test I have done with simple numpy
arrays results in properly freed memory on the engines.
> Running ‘import gc; gc.collect()’ does not seem to help.****
> So, I really want to restart the python process without having to kill,
> and restart the engine.
Restart *is* killing and restarting (the restart in the notebook/qtconsole
is literally: kill, wait, restart).
One of the things that I have long wanted to add to the engines is
KernelManager capabilities, so that they can be interrupted/restarted like
the regular Kernels.
We have a bit of degeneracy in our terminology/implementation, because what
we call a KernelManager is really two very different things: a
KernelManager, and the base KernelClient. There should only be one
KernelManager per kernel/engine, and then one KernelClient per frontend.
This would allow remote signalling/interrupting/restarting, which we
currently don't have, and the same changes would enable the KernelManager
to be used by Engines.
> ** **
> --Toby Burnett****
> IPython-User mailing list
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the IPython-User