[IPython-user] Fwd: Preparing for the 0.7.4 release

[Stefan van der Walt]

On Fri, Mar 23, 2007 at 01:37:19PM -0600, Fernando Perez wrote:
> It does look like the default python console does some extra work with
> regards to unicode internally.
> 
> Removing Stefan's patch (iplib.py, line 1980):
> 
>         try:
>             #line = raw_input_original(prompt).decode(sys.stdin.encoding)
>             line = raw_input_original(prompt)
> 
> makes the input exception go away:
> 
> In [1]: "והצ"
> Out[1]: '\xd7\x95\xd7\x94\xd7\xa6'
> 
> 
> So we're basically still stuck.

The exception occurs at a different point now.  If we replace the code
around the patch with the following:

        try:
            print "decoding using ", sys.stdin.encoding
            raw = raw_input_original(prompt)
            print "raw input: ", raw
            print "decoded as: ", raw.decode(sys.stdin.encoding)
            line = unicode(raw.decode(sys.stdin.encoding))

I see the following:

In [1]: "והצ"
raw input:  "והצ"
decoded as:  "והצ"
---------------------------------------------------------------------------
<type 'exceptions.UnicodeEncodeError'>    Traceback (most recent call last)

/home/stefan/work/scipy/ipython/trunk/IPython/iplib.py in multiline_prefilter(self, line, continue_prompt)
   2247 out = []
   2248 for l in line.rstrip('\n').split('\n'):
-> 2249 out.append(self._prefilter(l, continue_prompt))
   2250 return '\n'.join(out)
   2251 

/home/stefan/work/scipy/ipython/trunk/IPython/iplib.py in _prefilter(self, line, continue_prompt)
   2151 # Let's try to find if the input line is a magic fn
   2152 oinfo = None
-> 2153 if hasattr(self,'magic_'+iFun):
   2154 # WARNING: _ofind uses getattr(), so it can consume generators and
   2155 # cause other side effects.

<type 'exceptions.UnicodeEncodeError'>: 'ascii' codec can't encode characters in position 7-9: ordinal not in range(128)
decoding using  UTF-8

So the decoding itself looks to be fine.

The real problem is this:

hasattr("asd",'magic_'+u"והצ")

<type 'exceptions.UnicodeEncodeError'>: 'ascii' codec can't encode characters in position 6-8: ordinal not in range(128)

As far as I understand, strings need to be encoded in
'unicode_escape' before being used in Python source code, i.e.

hasattr("asd",'magic_'+u"והצ".encode('unicode_escape'))

works fine.

I don't suggest you apply the attached patch, but it serves to
demonstrate the workaround.

Cheers
Stéfan
-------------- next part --------------
Index: IPython/iplib.py
===================================================================
--- IPython/iplib.py	(revision 2175)
+++ IPython/iplib.py	(working copy)
@@ -1983,7 +1983,7 @@
         """
 
         try:
-            line = raw_input_original(prompt).decode(sys.stdin.encoding)
+            line = unicode(raw_input_original(prompt).decode(sys.stdin.encoding))
         except ValueError:
             warn("\n********\nYou or a %run:ed script called sys.stdin.close() or sys.stdout.close()!\nExiting IPython!")
             self.exit_now = True
@@ -2146,7 +2146,7 @@
 
         # Let's try to find if the input line is a magic fn
         oinfo = None
-        if hasattr(self,'magic_'+iFun):
+        if hasattr(self,'magic_'+iFun.encode('unicode_escape')):
             # WARNING: _ofind uses getattr(), so it can consume generators and
             # cause other side effects.
             oinfo = self._ofind(iFun) # FIXME - _ofind is part of Magic
Index: IPython/Magic.py
===================================================================
--- IPython/Magic.py	(revision 2175)
+++ IPython/Magic.py	(working copy)
@@ -232,7 +232,7 @@
         if not found:
             if oname.startswith(self.shell.ESC_MAGIC):
                 oname = oname[1:]
-            obj = getattr(self,'magic_'+oname,None)
+            obj = getattr(self,'magic_'+oname.encode('unicode_escape'),None)
             if obj is not None:
                 found = 1
                 ospace = 'IPython internal'