[IPython-dev] Bug caused by commit 2ccc90d68ac01145ab97

Jörgen Stenarson jorgen.stenarson@bostream...
Fri Mar 4 14:00:32 CST 2011


Thomas Kluyver skrev 2011-03-03 21:37:
> On 3 March 2011 19:22, Jörgen Stenarson <jorgen.stenarson@bostream.nu
> <mailto:jorgen.stenarson@bostream.nu>> wrote:
>
>     yes it is possible to enter other character codes but you have to
>     use the unicode functions in the api to read them properly. It is
>     for the 8-bit api that the encoding is valid.
>
>
>   OK, I'm not entirely following what you mean here. If I did a =
> raw_input(), and entered a non-cp1252 character, what ends up in a? Does
> it mysteriously turn the output into a unicode object? Or does it
> produce some odd string of bytes?
>
>  > In pyreadline we are using the wide api and are returning unicode
> strings, however I have not tested that extensively so there may be bugs
> in that handling.
>
> OK, does that mean that we should add unicode strings to add_history,
> rather than byte strings? Only I think readline in Python 2 on Linux
> choked on unicode when it contained non-ascii characters. Or is there an
> encoding we should use instead of cp1252?
>
> Thomas

Hi,

looking into it in more detail, pyreadline can return unicode strings 
but the call the hook that raw_input uses gets a regular string in 
return that has been encoded using .encode(pyreadline_codepage, u"replace").

I have attached a script and a screenshot of a session that demonstrates 
the difference between using raw_input and calling readline directly.

As things are now, using raw_input you should not receive any characters 
outside the current encoding because they will be converted to ? before 
being returned to raw_input.

/Jörgen
-------------- next part --------------
A non-text attachment was scrubbed...
Name: repl.png
Type: image/png
Size: 4196 bytes
Desc: not available
Url : http://mail.scipy.org/pipermail/ipython-dev/attachments/20110304/baa76356/attachment.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: repl.py
Type: application/x-python
Size: 122 bytes
Desc: not available
Url : http://mail.scipy.org/pipermail/ipython-dev/attachments/20110304/baa76356/attachment.bin 


More information about the IPython-dev mailing list