[IPython-user] IPython1 and TaskClient

Brian Granger ellisonbg.net@gmail....
Wed May 7 13:48:52 CDT 2008


> I developed my app using sync mode, it solved my problem!!!! I scaled out
> very easily! :D

Great

> Now I need just to finish the logging part and I would like to move
> everything to async mode.

What exactly do you mean by logging?

> So yes, I'm still stuck ( I must be honest and say that I didn't googled /
> read a lot trying to figure out how to move everything to async mode)
>
> BTW, is it a good idea to implement syslog logging on engine, to log it
> everything to a "central server"?

Can you give more info about what you mean by this?

Thanks

Brian

> My plan was to log to a local file per engine, and at the same time to a
> central server using syslog, but I still have to check what happens if
> syslog server dies.
>
>
>
> On Wed, Apr 23, 2008 at 8:22 PM, Brian Granger <ellisonbg.net@gmail.com>
> wrote:
>
> > Alberto,
> >
> > Sorry, I never got back to you on this.  Did you ever figure things
> > out.  Let me know - I have some time now and could help you get going
> > if you are still stuck.
> >
> > Brian
> >
> >
> >
> >
> > On Thu, Apr 10, 2008 at 3:35 AM, Alberto Mesas <amesas@e-commfactory.com>
> wrote:
> > > Thnx for the info! :)
> > >
> > >  I have been reading asynmultiengine1.py, but I got stuck.
> > >
> > >  from twisted.internet import reactor, defer
> > >  from ipython1.kernel import asynclient
> > >
> > >  client = asynclient.AsynTaskClient(('localhost', 10105))
> > >
> > >  def printer(r):
> > >     print r
> > >     return r
> > >
> > >
> > >  d = asynclient.AsynTask("a = %i"%n, pull="a")
> > >  #d = client.push(dict(a=5, b='asdf', c=[1,2,3]),targets=0,block=True)
> > >
> > >  d.addCallback(lambda _:
> client.pull(('a','b','c'),targets=0,block=True))
> > >  d.addBoth(printer)
> > >  d.addCallback(lambda _: reactor.stop())
> > >  reactor.run()
> > >
> > >
> > >
> > >  I'm sure I'm missing something, but can't get it :D
> > >
> > >  How I should rewrite the push/pull part?
> > >
> > >  BTW I have been testing my little app using SyncMode and a eight core
> > >  server and it worked flawlessly!!! Today I want to test using 8
> > >  servers and if it works and I can move it to Async, I will be ready to
> > >  setup it production environment :)
> > >
> > >  It just works!
> > >
> > >
> > >
> > >
> > >  On Thu, Apr 10, 2008 at 12:08 AM, Brian Granger
> <ellisonbg.net@gmail.com> wrote:
> > >  > > Newbie, just reading some doc :)
> > >  >  >
> > >  >  > Is it possible to pass functions to a TaskClient? I see it can be
> passed to
> > >  >  > an Engine, but I see no reference to it in the Task docs.
> > >  >
> > >  >  The way to do this is to also create a MultiEngineClient and use the
> > >  >  push_function method.  The key point is that you can use both the
> > >  >  TaskCliet and MultiEngineClient together and they talk to the same
> set
> > >  >  of engines underneath.  Thus, any functions you push using
> > >  >  MultiEngineClient will be available to code running in tasks (unless
> > >  >  you set the clearBefore or clearAfter flags in your Taks).
> > >  >
> > >  >  > And is there a way to setup some type of callback for a "finished"
> task? Or
> > >  >  > the only way to know the completed ones it to check queue_status?
> > >  >
> > >  >  There are two ways to accomplish this:
> > >  >
> > >  >  1) If you are using the standard client module
> > >  >  (ipython1.kernel.client) the TaskClient has a get_task_result method
> > >  >  that has the following signature:
> > >  >
> > >  >  get_task_result(self, taskid, block=False):
> > >  >
> > >  >  If you set block=True, this call will wait until the task finishes
> and
> > >  >  then return its result.
> > >  >
> > >  >  You can also wait for a set of tasks to be completed by using the
> > >  >  barrier method:
> > >  >
> > >  >  barrier(self, taskids)
> > >  >
> > >  >  here, taskids is simply a list or tuple of task ids.
> > >  >
> > >  >  2) If you really want a more event/callback driven interface, you
> can
> > >  >  use the asynclient module:
> > >  >
> > >  >  ipython1.kernel.asynclient
> > >  >
> > >  >  This has an AsynTaskClient that is designed to be used with
> Twisted's
> > >  >  even loop running.  Then all the methods return deferreds that you
> can
> > >  >  use the register callbacks on.
> > >  >
> > >  >  Here is an example of that:
> > >  >
> > >  >  docs/examples/asynmultiengine1.py
> > >  >
> > >  >  (OK, this is not an example of the Task stuff, but you will get the
> idea)..
> > >  >
> > >  >  Let us know if you have other questions.
> > >  >
> > >  >  Brian
> > >  >
> > >  >  > Thnx in advance for the info, and sorry if those are basic
> questions, just
> > >  >  > read some doc for evaluating IPython1 as the right  tool :)
> > >  >  >
> > >  >  > _______________________________________________
> > >  >  >  IPython-user mailing list
> > >  >  >  IPython-user@scipy.org
> > >  >  >  http://lists.ipython.scipy.org/mailman/listinfo/ipython-user
> > >  >  >
> > >  >  >
> > >  >
> > >
> >
>
>


More information about the IPython-user mailing list