[IPython-user] Persistence ipcontroller and ipengines

Franz Zieher franz.zieher@gmail....
Sat Feb 7 16:15:03 CST 2009



Vishal, thanks for the example that you provided. I did some reading on
twisted (should have
done that a while ago though). I maybe explain briefly the circumstances I
deploy 
the ipcontroller/ipengine setup.

I run MoinMoin (moin) with the twisted web server. 
In moin there is component called parser, that let's you
process a section on a page and output something else instead.

I developed a matplotlib parser that parses a python (matplotlib) code
section, sends the code section off to an ipengine, which generates 
charts, that are read by the parser and displayed instead of the 
matplotlib code. 

I did not wanted to run the matplotlib code directly by the web server,
because doing
so would obviously expose all the web server data to the user that is
building the
charting script and may possibly harm the running process.

I couldn't figure out how to run your example in context of the moin parser.
Instead I read through the IPython code and run the matplotlib code
(`script`)
in the following way.

        from IPython.kernel import asyncclient
        from twisted.internet.threads import blockingCallFromThread

        ... 

        # the tricky part of handling the execution on the client
        # this is valid when using twisted as web framework in moin
        tc  =
blockingCallFromThread(self.request.reactor,asyncclient.get_task_client,self.furl)
        tc  = tc.adapt_to_blocking_client()
        st  =
asyncclient.StringTask(script,clear_before=True,pull=pullObjs,push=pushObjs)
        tid = tc.run(st)
        res = tc.get_task_result(tid,block=True)

        if res.failure:
            self.failure = res.failure
            self.logdata = res.failure.getTraceback()

        if res.results.get('OUT',{}):
            self.outdata = res.results.get('OUT',{})

        # -------------------------------------------------------------

        ...

The key was that the moin request object has a reference to the reactor that
I've used to 
create a blocking task client, essentially the same that is created in
IPython.kernel.client

Franz





Brian Granger-2 wrote:
> 
> Thanks for the example Vishal.  Franz, let us know if you need other
> help getting this going.
> 
> But, the ipython engine and controller's should be able to run for a
> long time.  If they don't it is a bug and we should be able to fix it.
>  But with any distributed system, things *do* eventually go wrong
> (what if an ethernet cable get's unplugged) so if you really need the
> system to be bulletproof, you will have to add some real
> monitoring/fault tolerance to the setup.  But Vishal's month long
> running engines are a good sign!
> 
> 
> Cheers,
> 
> Brian
> 
> On Sat, Feb 7, 2009 at 7:03 AM, Vishal Vatsa <vishal.vatsa@gmail.com>
> wrote:
>> Hi Franz,
>>
>> So once you have your StringTask st,
>> you need an TaskClient, since you need to run
>> an asyncclient, (you will need to read up on twisted defered objects).
>>
>>
>> from IPython.kernel import asyncclient
>>
>> st = asyncclient.StringTask(some params)
>>
>> def submit_job(client):
>>    defered = client.run(st)
>>
>> defered_client = asyncclient.get_task_client()
>> defered_client.addCallback(submit_job)
>>
>>
>> Hope the above example makes sense. It does get a little bit complex
>> when dealing with asyncclient, as just about every thing is a defered.
>>
>> -vishal
>> 2009/2/7 Franz Zieher <franz.zieher@gmail.com>:
>>>
>>>
>>> Hi Vishal,
>>>
>>> thanks for the hint.
>>> As you pointed out, twisted is interesting.
>>> I tried to run my StringTask in the following way
>>>
>>> st = asyncclient.StringTask(script,pull=pullObjs,push=pushObjs)
>>> tc = asyncclient.get_task_client(self.furl)
>>> tid = tc.run(st)
>>> res = tc.get_task_result(tid,block=True)
>>>
>>> It turned out that tc does not have a run methon :-(
>>>
>>> Do you have an example on how I would run the above.
>>> Thanks
>>>
>>> Franz
>>>
>>>
>>>
>>>
>>>
>>> Vishal Vatsa-2 wrote:
>>>>
>>>> Hi
>>>>
>>>> I don't think that there is any major problems with
>>>> long running ipcontroller/engines, though i would
>>>> say call clear on you controller every so often.
>>>>
>>>> The longest run i have had so far is about a month.
>>>>
>>>> If you want to run a client from another reactor,
>>>> Use the asyncclient, twisted is interesting :)
>>>>
>>>> If you need more information, please ask.
>>>>
>>>> Regards,
>>>> -Vishal
>>>>
>>>> ----- Original message --
>>>> I have a couple of questions related to that procedure
>>>> - Is it wise to have long running ipcontroller/ipengines?
>>>> - Can the IPython.kernel.client be used within moin run by twisted (the
>>>> reactor is started by moin)?
>>>> - Are there any precautions (cleanups) that shall be performed before
>>>> launching the StringTask
>>>>
>>>> An early form of the parser can be found at
>>>> http://moinmo.in/ParserMarket/matplotlib
>>>>
>>>> Thanks for any thoughts on that.
>>>>
>>>> Franz
>>>> --
>>>> View this message in context:
>>>> http://www.nabble.com/Persistence-ipcontroller-and-ipengines-tp21800901p21800901.html
>>>> Sent from the IPython - User mailing list archive at Nabble.com.
>>>>
>>>> _______________________________________________
>>>> IPython-user mailing list
>>>> IPython-user@scipy.org
>>>> http://lists.ipython.scipy.org/mailman/listinfo/ipython-user
>>>>
>>>>
>>>> _______________________________________________
>>>> IPython-user mailing list
>>>> IPython-user@scipy.org
>>>> http://lists.ipython.scipy.org/mailman/listinfo/ipython-user
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Persistence-ipcontroller-and-ipengines-tp21800901p21888368.html
>>> Sent from the IPython - User mailing list archive at Nabble.com.
>>>
>>> _______________________________________________
>>> IPython-user mailing list
>>> IPython-user@scipy.org
>>> http://lists.ipython.scipy.org/mailman/listinfo/ipython-user
>>>
>> _______________________________________________
>> IPython-user mailing list
>> IPython-user@scipy.org
>> http://lists.ipython.scipy.org/mailman/listinfo/ipython-user
>>
> _______________________________________________
> IPython-user mailing list
> IPython-user@scipy.org
> http://lists.ipython.scipy.org/mailman/listinfo/ipython-user
> 
> 

-- 
View this message in context: http://www.nabble.com/Persistence-ipcontroller-and-ipengines-tp21800901p21893417.html
Sent from the IPython - User mailing list archive at Nabble.com.



More information about the IPython-user mailing list