[IPython-dev] ipython embedded in twisted, no threads

Jörgen Stenarson jorgen.stenarson@bostream...
Wed Apr 18 13:37:07 CDT 2007


Hi Stefan,

I have done some more experimenting. And now I think I have something 
resembling what you were after. If you replace the emacs.py file in 
pyreadline/modes/emacs.py with the attached version. Then running the 
async_test.py should give you a prompt where you can type away and at 
the same time the window title has a counter that is increased every 
second. Make sure you have a recent update from svn before replacing 
emacs.py.

If you have any suggestions for improving this interface I'm open for 
suggestions. I'm not sure on the best way to handle Ctrl-D and ctrl-C, 
especially ctrl-c is problematic since it can be caught by other parts 
of the program as well.

/Jörgen

Stefan Rank skrev:
> Hi Jörgen,
> 
> on 16.04.2007 20:16 Jörgen Stenarson said the following:
>> Hi Stefan,
>>
>> I'm not sure exactly what you are after (I don't use twisted myself).
> 
> I am not an expert either, I am afraid.
> 
>> But in pyreadline it is now fairly straightforward to add a new 
>> console class (there are currently two: the standard win32 and one for 
>> .NET/IronPython) which should implement the lowlevel keyboard and 
>> screen output routines. If this is what is needed I would be happy to 
>> accept patches and answer questions to help make it happen. We would 
>> probably also have to work out a more sofisticated method of choosing 
>> low-level console backend than what is used now.
> 
> I think what is needed for readline support in twisted is not a new 
> backend but an additional frontend/API.
> The 'problem' (read advantage ;-) with twisted is that it needs a 
> callback interface that does not block, as it wants to do the polling 
> for input events itself.
> 
> GNU readline has a callback interface that should be usable for that:
> http://cnswww.cns.cwru.edu/php/chet/readline/readline.html#SEC41
> (for triggering single character reading)
> Would this be possible with pyreadline?
> 
> cheers,
> stefan
> 
>> Stefan Rank skrev:
>>> Hi,
>>> [apologies if this arrives twice, posting via gmane didn't work - yet.]
>>>
>>> attached is an ugly first attempt at embedding IPython in a twisted
>>> application without using threads.
>>>
>>> twisted.internet.stdio and a LineReceiver implementation are responsible
>>> for getting input events.
>>> I had to subclass the IPython shell and rearrange code to
>>> create a non-blocking callback interface (splitting on the main,
>>> blocking, raw_input call).
>>>
>>> It's far from perfect: readline functionality is missing (no commandline
>>> navigation, history navigation, completion), and sometimes continuation
>>> lines eat the input counter (In[2] ... In[4]).
>>>
>>> Would IPython developers be interested in patches that allow a less ugly
>>> version of such a callback-friendly subclass? ;-)
>>> The basic idea: refactor the main loop (InteractiveShell.interact) to::
>>>
>>>    setup()
>>>    while ...:
>>>        promptforline()
>>>        raw_input
>>>        handleline()
>>>    teardown()
>>>
>>> (I think it might also be interesting for the frontend to the IPython
>>> parallel computing effort, which uses twisted.)
>>>
>>> cheers,
>>> stefan
>>>
>>>
>>> ------------------------------------------------------------------------
>>>
>>> '''Testing a way to embed IPython in a twisted app without using 
>>> threads.
>>> (seems to work on Py25/WinXP, twisted2.5 patched with #2157, 
>>> IPython0.8.0)
>>>
>>> twisted.internet.stdio is responsible for getting user input.
>>> An implementation of the LineReceiver protocol passes received lines
>>> to a custom subclass of the IPython shell (main raw_input call removed).
>>>
>>> No readline behaviour is available.
>>> (But IPython still sets up readline and uses it for colour printing.)
>>>
>>> TODO:
>>>     integrate readline functionality: two possibilities
>>>
>>>     1) Change pyreadline so that it can be called back by t.i.stdio for
>>>        all things except waiting for input (i.e. completion, key 
>>> bindings, ...)
>>>        This would mean no further changes to IPython, which would still
>>>        be using pyreadline directly.
>>>        (or use a GNU readline callback interface: a t.i.stdio-alike 
>>> would be
>>>        polling/selecting for input, but then call readline to 
>>> actually consume it.)
>>>     2) Build a readline compatible interface for t.i.stdio
>>>        (only the parts for setting up completeres, key bindings, ...)
>>>        and make the IPython subclass use that instead of pyreadline
>>>
>>> As there is a lot of pasted IPython code including comments, I added
>>> ##### multi-hash comments and
>>> docstrings explaining the rearrangements.
>>>
>>> :author: strank
>>> '''
>>>
>>> __docformat__ = "restructuredtext en"
>>>
>>> import sys
>>> import __builtin__
>>>
>>> from twisted.internet import reactor, stdio
>>> from twisted.protocols import basic
>>> from IPython.Shell import IPShellEmbed
>>> from IPython.iplib import InteractiveShell, ultraTB
>>> from IPython.ipmaker import make_IPython
>>>
>>>
>>> def main():
>>>     ###### this is what we want:
>>>     reactor.callWhenRunning(startTwistedStdioShell)
>>>     ###### because this blocks the reactor while IPython is active:
>>>     #reactor.callWhenRunning(startIPShell)
>>>     reactor.run()
>>>
>>>
>>> def startIPShell(args=None):
>>>     '''The non-twisted-friendly way of embedding IPython.'''
>>>     if args is None:
>>>         args = []
>>>     some_recognisable_local = 'WAGAWAGA' # gets exposed in Shell
>>>     ipshell = IPShellEmbed(args,
>>>                            banner='Standard IPython called from 
>>> twisted (blocking)',
>>>                            exit_msg='Leaving Interpreter, back to 
>>> program.')
>>>     ipshell('***Called from top level. '
>>>             'Hit Ctrl-D to exit interpreter and continue program.')
>>>     reactor.stop()
>>>
>>>
>>> def startTwistedStdioShell(args=None, local_ns=None):
>>>     '''On Windows, this only works with a patched twisted,
>>>     because twisted.internet.stdio in twisted2.5 is unix only,
>>>     see ticket #2157.
>>>     '''
>>>     if args is None:
>>>         args = []
>>>     ipshell = TwistedIPShellEmbed([],
>>>                                   banner='IPython embedded in a 
>>> Twisted application',
>>>                                   exit_msg='Leaving Interpreter, back 
>>> to program.')
>>>     sp = ShellProtocol(ipshell, '***Called from top level. ', 
>>> local_ns=local_ns)
>>>     #sp.setRawMode()
>>>     stdio.StandardIO(sp)
>>>
>>>
>>> class TwistedIPShellEmbed(IPShellEmbed):
>>>     '''Same as superclass, BUT:
>>>     __init__ is duplicated, with a single difference:
>>>         The actual IP shell_class is set to our own subclass.
>>>     __call__ is split into two methods for setup / teardown.
>>>     '''
>>>
>>>     def 
>>> __init__(self,argv=None,banner='',exit_msg=None,rc_override=None,
>>>                  user_ns=None):
>>>         """Note that argv here is a string, NOT a list."""
>>>         self.set_banner(banner)
>>>         self.set_exit_msg(exit_msg)
>>>         self.set_dummy_mode(0)
>>>         # sys.displayhook is a global, we need to save the user's 
>>> original
>>>         # Don't rely on __displayhook__, as the user may have changed 
>>> that.
>>>         self.sys_displayhook_ori = sys.displayhook
>>>         # save readline completer status
>>>         try:
>>>             #print 'Save completer',sys.ipcompleter  # dbg
>>>             self.sys_ipcompleter_ori = sys.ipcompleter
>>>         except:
>>>             pass # not nested with IPython
>>>         self.IP = make_IPython(argv,rc_override=rc_override,
>>>                                embedded=True,
>>>                                user_ns=user_ns,
>>>                                ###### HERE IS THE DIFFERENCE  -- 
>>> StefanRank
>>>                                shell_class=TwistedInteractiveShell)
>>>         # copy our own displayhook also
>>>         self.sys_displayhook_embed = sys.displayhook
>>>         # and leave the system's display hook clean
>>>         sys.displayhook = self.sys_displayhook_ori
>>>         # don't use the ipython crash handler so that user exceptions 
>>> aren't
>>>         # trapped
>>>         sys.excepthook = ultraTB.FormattedTB(color_scheme = 
>>> self.IP.rc.colors,
>>>                                              mode = self.IP.rc.xmode,
>>>                                              call_pdb = self.IP.rc.pdb)
>>>         self.restore_system_completer()
>>>
>>>     def 
>>> __call__(self,header='',local_ns=None,global_ns=None,dummy=None):
>>>         # Set global subsystems (display,completions) to our values
>>>         sys.displayhook = self.sys_displayhook_embed
>>>         if self.IP.has_readline:
>>>             self.IP.set_completer()
>>>         if self.banner and header:
>>>             format = '%s\n%s\n'
>>>         else:
>>>             format = '%s%s\n'
>>>         banner =  format % (self.banner,header)
>>>         # Call the embedding code with a stack depth of 1 so it can 
>>> skip over
>>>         # our call and get the original caller's namespaces.
>>>         ###### STOP HERE, THIS embed_mainloop DOESN'T BLOCK, RETURN 
>>> THE SHELL --StefanRank
>>>         return 
>>> self.IP.embed_mainloop(banner,local_ns,global_ns,stack_depth=1)
>>>
>>>     def teardown(self):
>>>         ###### call the code that would have been called at the end 
>>> of IP.embed_mainloop:
>>>         self.IP.teardown()
>>>         ###### This is the code formerly known as: the end of __call__:
>>>         if self.exit_msg:
>>>             print self.exit_msg
>>>         # Restore global systems (display, completion)
>>>         sys.displayhook = self.sys_displayhook_ori
>>>         self.restore_system_completer()
>>>
>>>
>>> class TwistedInteractiveShell(InteractiveShell):
>>>     '''Subclassing to remove raw_input call and provide a callback 
>>> interface.
>>>     Copy-pasted and split up code of embed_mainloop, interact, 
>>> raw_input.
>>>     '''
>>>
>>>     def 
>>> embed_mainloop(self,header='',local_ns=None,global_ns=None,stack_depth=0): 
>>>
>>>         # Get locals and globals from caller
>>>         if local_ns is None or global_ns is None:
>>>             call_frame = sys._getframe(stack_depth).f_back
>>>             if local_ns is None:
>>>                 local_ns = call_frame.f_locals
>>>             if global_ns is None:
>>>                 global_ns = call_frame.f_globals
>>>         self.user_global_ns = global_ns
>>>         local_varnames = local_ns.keys()
>>>         self.user_ns.update(local_ns)
>>>         if local_ns is None and global_ns is None:
>>>             self.user_global_ns.update(__main__.__dict__)
>>>         self.set_completer_frame()
>>>         self.add_builtins()
>>>         self.interact(header)
>>>         return self ###### needed for later calls, interact is now 
>>> non-blocking
>>>
>>>     def teardown(self):
>>>         '''This is the code formerly known as: the end of interact 
>>> and embed_mainloop.'''
>>>         ##### from interact:
>>>         # We are off again...
>>>         __builtin__.__dict__['__IPYTHON__active'] -= 1
>>>         ##### from embed_mainloop:
>>>         # now, purge out the user namespace from anything we might 
>>> have added
>>>         # from the caller's local namespace
>>>         ###### local_varnames not accessible any more,
>>>         ###### should probably be saved... (TODO)
>>>         ###delvar = self.user_ns.pop
>>>         ###for var in local_varnames:
>>>         ###    delvar(var,None)
>>>         # and clean builtins we may have overridden
>>>         self.clean_builtins()
>>>
>>>     def interact(self, banner=None):
>>>         ###### HERE IT GETS HAIRY: code copied from 
>>> self.interact(header)
>>>         if self.exit_now:
>>>             # batch run -> do not interact
>>>             return ###### This does not make any sense now, since we 
>>> are non-blocking
>>>         cprt = 'Type "copyright", "credits" or "license" for more 
>>> information.'
>>>         if banner is None:
>>>             self.write("Python %s on %s\n%s\n(%s)\n" %
>>>                        (sys.version, sys.platform, cprt,
>>>                         self.__class__.__name__))
>>>         else:
>>>             self.write(banner)
>>>         more = 0
>>>         # Mark activity in the builtins
>>>         __builtin__.__dict__['__IPYTHON__active'] += 1
>>>         self.promptForLine(more)
>>>         return self
>>>
>>>     def promptForLine(self, more):
>>>         '''New method. To be called before waiting for the next line.'''
>>>         prompt = ''
>>>         ###### THIS NEEDED TO BE UNTANGLED: originally a while loop 
>>> in self.interact
>>>         # exit_now is set by a call to %Exit or %Quit
>>>         if not self.exit_now:
>>>             if more:
>>>                 prompt = self.hooks.generate_prompt(True)
>>>                 if self.autoindent:
>>>                     self.readline_startup_hook(self.pre_readline)
>>>             else:
>>>                 prompt = self.hooks.generate_prompt(False)
>>>             ####### code from self.raw_input:
>>>             self.set_completer()
>>>             ####### write the prompt here, writing it in the 
>>> LineReceiver
>>>             ####### with sendLine messes up the colouring
>>>             self.write(prompt)
>>>         ####### raw_input BEGONE!!!
>>>         ####### unfortunately all the exception handling in 
>>> self.interact is also gone.
>>>         ####### return and wait for a call to useLine
>>>
>>>     def useLine(self, line, continue_prompt=False):
>>>         '''New method. To be called with a new received line.'''
>>>         ####### code from self.raw_input:
>>>         if line.strip():
>>>             if continue_prompt:
>>>                 self.input_hist_raw[-1] += '%s\n' % line
>>>                 if self.has_readline: # and some config option is set?
>>>                     try:
>>>                         histlen = 
>>> self.readline.get_current_history_length()
>>>                         newhist = self.input_hist_raw[-1].rstrip()
>>>                         self.readline.remove_history_item(histlen-1)
>>>                         
>>> self.readline.replace_history_item(histlen-2,newhist)
>>>                     except AttributeError:
>>>                         pass # re{move,place}_history_item are new in 
>>> 2.4.
>>>             else:
>>>                 self.input_hist_raw.append('%s\n' % line)
>>>         try:
>>>             lineout = self.prefilter(line,continue_prompt)
>>>         except:
>>>             # blanket except, in case a user-defined prefilter 
>>> crashes, so it
>>>             # can't take all of ipython with it.
>>>             self.showtraceback()
>>>             lineout = ''
>>>         ###### CODE FROM INTERACT AGAIN:
>>>         more = self.push(lineout)
>>>         if (self.SyntaxTB.last_syntax_error and
>>>             self.rc.autoedit_syntax):
>>>             self.edit_syntax_error()
>>>         self.promptForLine(more)
>>>         ###### indicate if we want to quit
>>>         return not self.exit_now
>>>
>>>
>>> class ShellProtocol(basic.LineReceiver):
>>>     delimiter = '\n' # does not work with '\r\n' on Windows...
>>>                      # this might be an issue with the #2157 patches?
>>>
>>>     def __init__(self, ipshell, banner, **kwargs):
>>>         self.ipshell = ipshell
>>>         self.banner = banner
>>>         self.kwargs = kwargs
>>>
>>>     def connectionMade(self):
>>>         #self.sendLine("Yay, type something!")
>>>         # locals here are available in the shell:
>>>         # (but in practice the locals of the ShellProtocol() call site
>>>         # would be more interesting.)
>>>         another_strange_local = 'hubabuba'
>>>         from twisted.internet import reactor
>>>         self.sheller = self.ipshell(self.banner
>>>                 + '\n  quit.....quits via LineReceiver,'
>>>                 + '\n  exit()...quits via IPython,'
>>>                 + '\n  pause....test pausing the LineReceiver',
>>>                 **self.kwargs)
>>>
>>>     def lineReceived(self, line):
>>>         if line == 'quit':
>>>             self.sendLine('OK quitting')
>>>             self.transport.loseConnection()
>>>             return
>>>         if line == 'pause':
>>>             self.sendLine('OK pausing for 2 seconds')
>>>             self.pauseProducing()
>>>             self.clearLineBuffer()
>>>             # still receives keys typed during the pause,
>>>             # no matter how many clearLineBuffer()s, I use.
>>>             # might be a feature of Windows cmd... ?
>>>             def restart():
>>>                 self.clearLineBuffer()
>>>                 self.resumeProducing()
>>>                 self.clearLineBuffer()
>>>             reactor.callLater(2, restart)
>>>             return
>>>         #self.sendLine("Echo: " + line)
>>>         if not self.sheller.useLine(line): # processes line, prints 
>>> output and next prompt
>>>             self.sendLine("IPython wants to quit")
>>>             self.transport.loseConnection()
>>>         #self.transport.write("Yay, type more!")
>>>
>>>     #def rawDataReceived(self, data):
>>>     #    self.sendLine('RawEcho: ' + str(data))
>>>     #    self.transport.write("Yay, type more!")
>>>
>>>     def connectionLost(self, reason):
>>>         try:
>>>             self.ipshell.teardown()
>>>         except IOError:
>>>             pass # is raised by IPython code when you type quit
>>>         # stop the reactor, only because this is meant to be run in 
>>> Stdio.
>>>         reactor.stop()
>>>
>>>
>>> if __name__ == '__main__':
>>>     main()
>>>
>>>
>>> ------------------------------------------------------------------------
>>>
>>> _______________________________________________
>>> IPython-dev mailing list
>>> IPython-dev@scipy.org
>>> http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev
>>
>> _______________________________________________
>> IPython-dev mailing list
>> IPython-dev@scipy.org
>> http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev
> 

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: async_test.py
Url: http://lists.ipython.scipy.org/pipermail/ipython-dev/attachments/20070418/5e852315/attachment-0002.pl 
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: emacs.py
Url: http://lists.ipython.scipy.org/pipermail/ipython-dev/attachments/20070418/5e852315/attachment-0003.pl 


More information about the IPython-dev mailing list