Python Unicode oddness

If you have a Python string with an invalid character for the current character set, then other characters may get removed unexpectedly. For instance:

>>> a = 'FOO\xe0BAR'
>>> print '%r' % a
'FOO\xe0BAR'

>>> print '%r' % unicode(a, 'utf8')
Traceback (most recent call last):
 File "", line 1, in 
 File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/encodings/utf_8.py", line 16, in decode
   return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 3-5: invalid data

Fair enough, \xe0 isn't a valid utf8 character. But if I tell the decoder to just ignore any characters it doesn't understand, it also eats the "B" and "A" characters!

>>> print '%r' % unicode(a, 'utf8', 'replace')
u'FOO\ufffdR'
>>> print '%r' % unicode(a, 'utf8', 'ignore')
u'FOOR'

That was unexpected. One day I'm sure someone will explain why it happens to me.

The February issue of Python Magazine is out and contains my article "Using Python and SOAP to create a CLI for JIRA" about the Python CLI that I wrote for JIRA a while ago. The article's summary reads:

Many web applications include an API that lets you interact with them from the command line as well as with a browser. In this article, Matthew shows how to build a command line interface for JIRA, a well-known issue tracking system, using Python and SOAP. JIRA is a Java application, but using SOAP allows you access to many of its features using just Python.

One hint that I wish I had remembered to add is that when you have a redirect to your JIRA server, for instance when http://jira.mycompany.com is redirected by Apache or IIS to http://jira.mycompany.com:8080, you may see your login mysteriously fail. The answer is that you have to use the redirected URL with the JIRA CLI. You can find out what the redirected URL is by running the CLI with the argument -v10 to increase the logging verbosity and look at the line that starts with "Host". This example shows that the port to use is 8080.

*** Outgoing HTTP headers **********************************************
POST /rpc/soap/jirasoapservice-v2 HTTP/1.0
Host: localhost:8080