[IPython-user] TaskClient doesn't work after importing a module in MultiEngineClient

Brian Granger ellisonbg.net@gmail....
Wed Nov 19 22:26:15 CST 2008

On Mon, Nov 17, 2008 at 2:16 AM, Pecevski Dejan <dejan@igi.tugraz.at> wrote:
> Hi,
> Thanks for your reply.
> Brian Granger wrote:
>> I haven't look in detail at exactly what the tests script is doing,
>> but here is the problem:
>> You are doing something that causes the twisted reactor to be started
>> twice.  The reactor is started for the controller and engine
>> automatically when they start and for the client when you import
>> IPython.kernel.client.  The problem is that when you execute the
>> script in the engine, the engine already has the reactor running, but
>> at the top of the script you have the import of IPython.kernel.client,
>> which tries to start the reactor again.
> You are right, this causes the problem. I've put the import of
> Ipython.kernel.client
> under
> if __name__ == '__main__':
> in the test script, and it works OK.
> But shouldn't it be possible to invoke multiple times the  import
> operation of Ipython.kernel.client?

You _can_ import IPython.kernel.client multiple times, however, that
is not what you are doing.  Instead you are importing this *inside* a
running ipython engine.

> Here is where the importing in the engines arises in my code.
> My program is composed of two scripts, e.g. myclient.py and mytask.py
> The client script is my main program, and the mytask.py script is the code
> that will run on the engines.

This is a good and common design pattern to use with IPython.kernel.

> I have also my own module with common utilities related to the stuff I'm
> doing,
> MyUtils.py. In utils.py there is also my own custom Cluster class which
> encapsulates
> some operations with the ipcluster, starting, stopping, killing engines
> etc.

Nice!  I don't know if you are interested, but I just finished the
first phase of a re-write of ipcluster.  It is greatly improved in
many respects, but there is still a lot of work to do.  Would you be
willing to share your custom Cluster class with us.  I am not how it
would fit into our design, but we could have a look and see if we
could use it.  We _really_ want to improve the managing of the
engines.  My branch on launchpad has the refactored ipcluster (it has
not been merged into trunk, but will be soon).


> I import MyUtils both in the client script and in the task script, in
> the client to
> start and connect to the cluster and in the engines to use some other
> functionalities in MyUtils
> that are not related to the cluster.
> If I want to avoid the problem, then probably I should split MyUtils in
> two modules, let's say
> MyClusterUtils and MyOtherUtils, and import MyClusterUtils only on the
> client.

Yes, splitting this up like this would help the problem.

> But sometimes it might be necessary to use one of the engines as a
> client connected to another ipcontroller,
> and it seems impossible to do it, if one cannot import the client in one
> of the engines.

Yes, this should totally work and I consider it a bug that it doesn't.
 Could you file a bug report on our launchpad site:




> Cheers,
> Dejan
>> Cheers,
>> Brian
> _______________________________________________
> IPython-user mailing list
> IPython-user@scipy.org
> http://lists.ipython.scipy.org/mailman/listinfo/ipython-user

More information about the IPython-user mailing list