[IPython-user] Several problems with cp720 (Arabic Windows-1256)

Jörgen Stenarson jorgen.stenarson@bostream...
Fri Jan 4 11:53:28 CST 2008


Diaa Sami skrev:
> Hi,
> I'm having problems with IPython with all versions after 0.7.3, and I just found that there are
> problems with the most recent pyreadline too.
> I'm on an English Windows XP system with Arabic language support(which means that
> sys.stdin.encoding='cp720'), Control Panel > Regional and Language options > Advanced tab > Arabic
> (Egypt) selected.
> You can find the problematic cases below
> 1- I upgraded to a recent IPython(0.8.1) but I left the old pyreadline-1.3, IPython starts properly
> but whenever I enter any text I get the following error:
> 
Diaa,

The problem is that current versions of python does not support cp720. 
After some googling it seems there are fixes in the trunk for python but 
that doesn't help us.

For me (using swedish windows) the default codepage of the cmd.exe is 
cp850. But there is a mismatch to the encoding used in the filesystem 
which makes it more convenient for me to always launch cmd.exe such that 
it switches to cp1252 using the command chcp 1252. Perhaps you could try 
to do the same but with chcp 1256 to get arabic.

You could also try adding:
import pyreadline.unicode_helper
pyreadline.unicode_helper.pyreadline_codepage="cp1256"

to force pyreadline to use another codepage than the default, however 
this may not work if there is a mismatch between 1256 and 720.

Please let us know how it goes

/Jörgen


> ---- start here
> In [1]: f
> ---------------------------------------------------------------------------
> LookupError                               Traceback (most recent call last)
> 
> C:\Python25\Lib\site-packages\IPython\iplib.py in raw_input(self, prompt, contin
> ue_prompt)
>     2049
>     2050         try:
> -> 2051             line = raw_input_original(prompt).decode(self.stdin_encoding
> )
>     2052         except ValueError:
>     2053             warn("\n********\nYou or a %run:ed script called sys.stdin.c
> lose()"
> 
> LookupError: unknown encoding: cp720
> ---- end here
> 
> I solved this problem by changing line 380 in iplib.py to
> self.stdin_encoding = 'ascii'
> instead of
> self.stdin_encoding = sys.stdin.encoding or 'ascii'
> 
> 2- I upgraded to the current IPython(0.8.2) and using the latest pyreadline-1.5, when starting
> IPython I get the following error:
> 
> ---- start here
> ---------------------------------------------------------------------------
> LookupError                            Python 2.5.1: C:\Python25\python.exe
>                                                     Mon Dec 24 16:33:25 2007
> A problem occured executing Python code.  Here is the sequence of function
> calls leading up to the error, with the most recent (innermost) call last.
> 
> C:\Python25\scripts\ipython in <module>()
>       12 IPython.Shell.IPShell().mainloop(sys_exit=1)
>       13
>       14 [or simply IPython.Shell.IPShell().mainloop(1) ]
>       15
>       16 and IPython will be your working environment when you start python. The
> final
>       17 sys.exit() call will make python exit transparently when IPython finishe
> s, so
>       18 you don't have an extra prompt to get out of.
>       19
>       20 This is probably useful to developers who manage multiple Python version
> s and
>       21 don't want to have correspondingly multiple IPython versions. Note that
> in
>       22 this mode, there is no way to pass IPython any command-line options, as
> those
>       23 are trapped first by Python itself.
>       24 """
>       25
>       26 import IPython
> ---> 27 IPython.Shell.start().mainloop()
>          global IPython.Shell.start.mainloop = undefined
>       28
>       29
>       30
>       31
>       32
>       33
>       34
>       35
>       36
>       37
>       38
>       39
>       40
>       41
>       42
> 
> C:\Python25\lib\site-packages\IPython\Shell.py in start(user_ns=None)
>     1137         arg1 = argv[1]
>     1138         if arg1.endswith('-gthread'):
>     1139             shell = IPShellGTK
>     1140         elif arg1.endswith( '-qthread' ):
>     1141             shell = IPShellQt
>     1142         elif arg1.endswith( '-q4thread' ):
>     1143             shell = IPShellQt4
>     1144         elif arg1.endswith('-wthread'):
>     1145             shell = IPShellWX
>     1146         elif arg1.endswith('-pylab'):
>     1147             shell = _matplotlib_shell_class()
>     1148         else:
>     1149             shell = IPShell
>     1150     else:
>     1151         shell = IPShell
> -> 1152     return shell(user_ns = user_ns)
>          shell = <class IPython.Shell.IPShell at 0x00FB6990>
>          user_ns = None
>     1153
>     1154 # Some aliases for backwards compatibility
>     1155 IPythonShell = IPShell
>     1156 IPythonShellEmbed = IPShellEmbed
>     1157 #************************ End of file <Shell.py> ***********************
> ****
>     1158
>     1159
>     1160
>     1161
>     1162
>     1163
>     1164
>     1165
>     1166
>     1167
> 
> C:\Python25\lib\site-packages\IPython\Shell.py in __init__(self=<IPython.Shell.I
> PShell instance at 0x00AF3F80>, argv=None, user_ns=None, user_global_ns=None, de
> bug=1, shell_class=<class 'IPython.iplib.InteractiveShell'>)
>       59 # Tag when runcode() is active, for exception handling
>       60 CODE_RUN = None
>       61
>       62 #-----------------------------------------------------------------------
> ------
>       63 # This class is trivial now, but I want to have it in to publish a clean
> 
>       64 # interface. Later when the internals are reorganized, code that uses th
> is
>       65 # shouldn't have to change.
>       66
>       67 class IPShell:
>       68     """Create an IPython instance."""
>       69
>       70     def __init__(self,argv=None,user_ns=None,user_global_ns=None,
>       71                  debug=1,shell_class=InteractiveShell):
>       72         self.IP = make_IPython(argv,user_ns=user_ns,
>       73                                user_global_ns=user_global_ns,
> ---> 74                                debug=debug,shell_class=shell_class)
>          debug = 1
>          shell_class = <class 'IPython.iplib.InteractiveShell'>
>       75
>       76     def mainloop(self,sys_exit=0,banner=None):
>       77         self.IP.mainloop(banner)
>       78         if sys_exit:
>       79             sys.exit()
>       80
>       81 #-----------------------------------------------------------------------
> ------
>       82 def kill_embedded(self,parameter_s=''):
>       83     """%kill_embedded : deactivate for good the current embedded IPython
> .
>       84
>       85     This function (after asking for confirmation) sets an internal flag
> so that
>       86     an embedded IPython will never activate again.  This is useful to
>       87     permanently disable a shell that is being called inside a loop: once
>   you've
>       88     figured out what you needed from it, you may then kill it and the pr
> ogram
>       89     will then continue to run without the interactive shell interfering
> again.
> 
> C:\Python25\Lib\site-packages\IPython\ipmaker.py in make_IPython(argv=[r'C:\Pyth
> on25\scripts\ipython'], user_ns=None, user_global_ns=None, debug=1, rc_override=
> None, shell_class=<class 'IPython.iplib.InteractiveShell'>, embedded=False, **kw
> ={})
>      745         IP_rc.banner = 0
>      746     if IP_rc.banner:
>      747         BANN_P = IP.BANNER_PARTS
>      748     else:
>      749         BANN_P = []
>      750
>      751     if IP_rc.profile: BANN_P.append('IPython profile: %s\n' % IP_rc.prof
> ile)
>      752
>      753     # add message log (possibly empty)
>      754     if msg.summary: BANN_P.append(msg.summary)
>      755     # Final banner is a string
>      756     IP.BANNER = '\n'.join(BANN_P)
>      757
>      758     # Finalize the IPython instance.  This assumes the rc structure is f
> ully
>      759     # in place.
> --> 760     IP.post_config_initialization()
>          IP.post_config_initialization = <bound method InteractiveShell.post_conf
> ig_initialization of <IPython.iplib.InteractiveShell object at 0x00AF5750>>
>      761
>      762     return IP
>      763 #************************ end of file <ipmaker.py> *********************
> *****
>      764
>      765
>      766
>      767
>      768
>      769
>      770
>      771
>      772
>      773
>      774
>      775
> 
> C:\Python25\Lib\site-packages\IPython\iplib.py in post_config_initialization(sel
> f=<IPython.iplib.InteractiveShell object at 0x00AF5750>)
>      650         This is called after the configuration files have been processed
>   to
>      651         'finalize' the initialization."""
>      652
>      653         rc = self.rc
>      654
>      655         # Object inspector
>      656         self.inspector = OInspect.Inspector(OInspect.InspectColors,
>      657                                             PyColorize.ANSICodeColors,
>      658                                             'NoColor',
>      659                                             rc.object_info_string_level)
> 
>      660
>      661         self.rl_next_input = None
>      662         self.rl_do_indent = False
>      663         # Load readline proper
>      664         if rc.readline:
> --> 665             self.init_readline()
>          self.init_readline = <bound method InteractiveShell.init_readline of <IP
> ython.iplib.InteractiveShell object at 0x00AF5750>>
>      666
>      667
>      668         # local shortcut, this is used a LOT
>      669         self.log = self.logger.log
>      670
>      671         # Initialize cache, set in/out prompts and printing system
>      672         self.outputcache = CachedOutput(self,
>      673                                         rc.cache_size,
>      674                                         rc.pprint,
>      675                                         input_sep = rc.separate_in,
>      676                                         output_sep = rc.separate_out,
>      677                                         output_sep2 = rc.separate_out2,
>      678                                         ps1 = rc.prompt_in1,
>      679                                         ps2 = rc.prompt_in2,
>      680                                         ps_out = rc.prompt_out,
> 
> C:\Python25\Lib\site-packages\IPython\iplib.py in init_readline(self=<IPython.ip
> lib.InteractiveShell object at 0x00AF5750>)
>     1348             # is being used (as on Leopard) the readline config is
>     1349             # not run as the syntax for libedit is different.
>     1350             if not readline.uses_libedit:
>     1351                 for rlcommand in self.rc.readline_parse_and_bind:
>     1352                     readline.parse_and_bind(rlcommand)
>     1353
>     1354             # remove some chars from the delimiters list
>     1355             delims = readline.get_completer_delims()
>     1356             delims = delims.translate(string._idmap,
>     1357                                       self.rc.readline_remove_delims)
>     1358             readline.set_completer_delims(delims)
>     1359             # otherwise we end up with a monster history after a while:
>     1360             readline.set_history_length(1000)
>     1361             try:
>     1362                 #print '*** Reading readline history'  # dbg
> -> 1363                 readline.read_history_file(self.histfile)
>          readline.read_history_file = <bound method Readline.read_history_file of
>   <pyreadline.rlmain.Readline object at 0x00E73A90>>
>          self.histfile = 'C:\\Documents and Settings\\Administrator\\_ipython\\hi
> story'
>     1364             except IOError:
>     1365                 pass  # It doesn't exist yet.
>     1366
>     1367             atexit.register(self.atexit_operations)
>     1368             del atexit
>     1369
>     1370         # Configure auto-indent for all platforms
>     1371         self.set_autoindent(self.rc.autoindent)
>     1372
>     1373     def ask_yes_no(self,prompt,default=True):
>     1374         if self.rc.quiet:
>     1375             return True
>     1376         return ask_yes_no(prompt,default)
>     1377
>     1378     def _should_recompile(self,e):
> 
> C:\Python25\Lib\site-packages\pyreadline\rlmain.py in read_history_file(self=<py
> readline.rlmain.Readline object at 0x00E73A90>, filename=r'C:\Documents and Sett
> ings\Administrator\_ipython\history')
>      168
>      169     def set_history_length(self, length):
>      170         '''Set the number of lines to save in the history file.
>      171
>      172         write_history_file() uses this value to truncate the history fil
> e
>      173         when saving. Negative values imply unlimited history file size.
>      174         '''
>      175         self._history.set_history_length(length)
>      176
>      177     def clear_history(self):
>      178         '''Clear readline history'''
>      179         self._history.clear_history()
>      180
>      181     def read_history_file(self, filename=None):
>      182         '''Load a readline history file. The default filename is ~/.hist
> ory.'''
> --> 183         self._history.read_history_file(filename)
>          self._history.read_history_file = <bound method LineHistory.read_history
> _file of <pyreadline.lineeditor.history.LineHistory object at 0x00E73BD0>>
>          filename = 'C:\\Documents and Settings\\Administrator\\_ipython\\history
> '
>      184
>      185     def write_history_file(self, filename=None):
>      186         '''Save a readline history file. The default filename is ~/.hist
> ory.'''
>      187         self._history.write_history_file(filename)
>      188
>      189     #Completer functions
>      190
>      191     def set_completer(self, function=None):
>      192         '''Set or remove the completer function.
>      193
>      194         If function is specified, it will be used as the new completer
>      195         function; if omitted or None, any completer function already
>      196         installed is removed. The completer function is called as
>      197         function(text, state), for state in 0, 1, 2, ..., until it retur
> ns a
>      198         non-string value. It should return the next possible completion
> 
> C:\Python25\Lib\site-packages\pyreadline\lineeditor\history.py in read_history_f
> ile(self=<pyreadline.lineeditor.history.LineHistory object at 0x00E73BD0>, filen
> ame=r'C:\Documents and Settings\Administrator\_ipython\history')
>       55
>       56     history_length=property(get_history_length,set_history_length)
>       57     history_cursor=property(get_history_cursor,set_history_cursor)
>       58
>       59     def clear_history(self):
>       60         '''Clear readline history.'''
>       61         self.history[:] = []
>       62         self.history_cursor = 0
>       63
>       64     def read_history_file(self, filename=None):
>       65         '''Load a readline history file.'''
>       66         if filename is None:
>       67             filename=self.history_filename
>       68         try:
>       69             for line in open(filename, 'r'):
> ---> 70                 self.add_history(lineobj.ReadLineTextBuffer(ensure_unico
> de(line.rstrip())))
>          self.add_history = <bound method LineHistory.add_history of <pyreadline.
> lineeditor.history.LineHistory object at 0x00E73BD0>>
>          global lineobj.ReadLineTextBuffer = <class 'pyreadline.lineeditor.lineob
> j.ReadLineTextBuffer'>
>          global ensure_unicode = <function ensure_unicode at 0x00C91630>
>          line.rstrip = <built-in method rstrip of str object at 0x00FE6908>
>       71         except IOError:
>       72             self.history = []
>       73             self.history_cursor = 0
>       74
>       75     def write_history_file(self, filename=None):
>       76         '''Save a readline history file.'''
>       77         if filename is None:
>       78             filename=self.history_filename
>       79         fp = open(filename, 'wb')
>       80         for line in self.history[-self.history_length:]:
>       81             fp.write(ensure_str(line.get_line_text()))
>       82             fp.write('\n')
>       83         fp.close()
>       84
>       85
> 
> C:\Python25\Lib\site-packages\pyreadline\unicode_helper.py in ensure_unicode(tex
> t='i1, i2=6,26')
>        5 #  Distributed under the terms of the BSD License.  The full license is
> in
>        6 #  the file COPYING, distributed as part of this software.
>        7 #***********************************************************************
> ******
>        8 import sys
>        9
>       10 try:
>       11     pyreadline_codepage=sys.stdout.encoding
>       12 except AttributeError:        #This error occurs when pdb imports readli
> ne and doctest has replaced
>       13                               #stdout with stdout collector
>       14     pyreadline_codepage="ascii"   #assume ascii codepage
>       15
>       16
>       17 def ensure_unicode(text):
>       18     """helper to ensure that text passed to WriteConsoleW is unicode"""
>       19     if isinstance(text, str):
> ---> 20         return text.decode(pyreadline_codepage, "replace")
>          text.decode = <built-in method decode of str object at 0x00FE6DB8>
>          global pyreadline_codepage = 'cp720'
>       21     return text
>       22
>       23 def ensure_str(text):
>       24     """Convert unicode to str using pyreadline_codepage"""
>       25     if isinstance(text, unicode):
>       26         return text.encode(pyreadline_codepage, "replace")
>       27     return text
>       28
>       29
>       30
>       31
>       32
>       33
>       34
>       35
> 
> LookupError: unknown encoding: cp720
> 
> **********************************************************************
> 
> Oops, IPython crashed. We do our best to make it stable, but...
> 
> A crash report was automatically generated with the following information:
>    - A verbatim copy of the crash traceback.
>    - A copy of your input history during this session.
>    - Data on your current IPython configuration.
> 
> It was left in the file named:
>          'C:\Documents and Settings\Administrator\_ipython\IPython_crash_report.t
> xt'
> If you can email this file to the developers, the information in it will help
> them in understanding and correcting the problem.
> 
> You can mail it to: Ville Vainio at vivainio@gmail.com
> with the subject 'IPython Crash Report'.
> 
> If you want to do it now, the following command will work (under Unix):
> mail -s 'IPython Crash Report' vivainio@gmail.com < C:\Documents and Settings\Ad
> ministrator\_ipython\IPython_crash_report.txt
> 
> To ensure accurate tracking of this issue, please file a report about it at:
> http://projects.scipy.org/ipython/ipython/report
> ---- end here
> 
> I solved this error by installing pyreadline-1.4.4 instead.
> 
> So apparently the first error is a problem with IPython itself but the latter is a problem with
> pyreadline.
> 
> I'd appreciate if these problem can be solved for future versions of IPython(without having to
> modify it manually), I'm ready to do some testing if needed.
> 
> Thanks
> 
> Regards,
> Diaa M. Sami
> 
> _______________________________________________
> IPython-user mailing list
> IPython-user@scipy.org
> http://lists.ipython.scipy.org/mailman/listinfo/ipython-user
> 



More information about the IPython-user mailing list