unicode - Best Practices for Python UnicodeDecodeError

Question

I use Pylons framework, Mako template for a web based application. I wasn't really bother too deep into the way Python handles the unicode strings. I had tense moment when I did see my site crash when the page is rendered and later I came to know that it was related to UnicodeDecodeError.

After seeing the error, I started mesh around my Python code adding encode, decode calls for string with 'ignore' option but still I could not see the errors gone sometime.

Finally I used to decode to ascii with ignore and made the site running without any crash.

Input to my site comes through many sites. This means that I do not control the languages or language of choice. My site supports international languages and along with English. I have feed aggregation which generally not bother about unicode/ascii/utf-8. While I display the text through mako template, I display as it is.

Not being a web expert, what are the best practices to handle the strings within the Python project? Should I care only while rendering the text or all the phase of the application?

Answer 1

If you have influence on it, this is the painless way:

• know your input encoding (or decode with ignore) and decode(encoding) the data as soon as it hits your app
• work internally only with unicode (u'something' is unicode), also in the database
• for rendering, export etc, anytime it leaves your app, encode('utf-8') the data