python - latin-1 to ascii

Question

I have a unicode string with accented latin chars e.g.

n=unicode('Wikipédia, le projet d’encyclopédie','utf-8')

I want to convert it to plain ascii i.e. 'Wikipedia, le projet dencyclopedie', so all acute/accent,cedilla etc should get removed

What is the fastest way to do that, as it needed to be done for matching a long autocomplete dropdown list

Conclusion: As one my criteria is speed, Lennart's 'register your own error handler for unicode encoding/decoding' gives best result (see Alex's answer), speed difference increases further as more and more chars are latin.

Here is the translation table I am using, also modified error handler as it need to take care of whole range of un-encoded char from error.start to error.end

# -*- coding: utf-8 -*-
import codecs

"""
This is more of visual translation also avoiding multiple char translation
e.g. ￡ may be written as {pound}
"""
latin_dict = {
u"?": u"!", u"￠": u"c", u"￡": u"L", u"¤": u"o", u"￥": u"Y",
u"|": u"|", u"§": u"S", u"¨": u"`", u"?": u"c", u"a": u"a",
u"?": u"<<", u"?": u"-", u"-": u"-", u"?": u"R", u"ˉ": u"-",
u"°": u"o", u"±": u"+-", u"2": u"2", u"3": u"3", u"′": u"'",
u"μ": u"u", u"?": u"P", u"·": u".", u"?": u",", u"1": u"1",
u"o": u"o", u"?": u">>", u"?": u"1/4", u"?": u"1/2", u"?": u"3/4",
u"?": u"?", u"à": u"A", u"á": u"A", u"?": u"A", u"?": u"A",
u"?": u"A", u"?": u"A", u"?": u"Ae", u"?": u"C", u"è": u"E",
u"é": u"E", u"ê": u"E", u"?": u"E", u"ì": u"I", u"í": u"I",
u"?": u"I", u"?": u"I", u"D": u"D", u"?": u"N", u"ò": u"O",
u"ó": u"O", u"?": u"O", u"?": u"O", u"?": u"O", u"×": u"*",
u"?": u"O", u"ù": u"U", u"ú": u"U", u"?": u"U", u"ü": u"U",
u"Y": u"Y", u"T": u"p", u"?": u"b", u"à": u"a", u"á": u"a",
u"a": u"a", u"?": u"a", u"?": u"a", u"?": u"a", u"?": u"ae",
u"?": u"c", u"è": u"e", u"é": u"e", u"ê": u"e", u"?": u"e",
u"ì": u"i", u"í": u"i", u"?": u"i", u"?": u"i", u"e": u"d",
u"?": u"n", u"ò": u"o", u"ó": u"o", u"?": u"o", u"?": u"o",
u"?": u"o", u"÷": u"/", u"?": u"o", u"ù": u"u", u"ú": u"u",
u"?": u"u", u"ü": u"u", u"y": u"y", u"t": u"p", u"?": u"y", 
u"’":u"'"}

def latin2ascii(error):
    """
    error is  protion of text from start to end, we just convert first
    hence return error.start+1 instead of error.end
    """
    return latin_dict[error.object[error.start]], error.start+1

codecs.register_error('latin2ascii', latin2ascii)

if __name__ == "__main__":
    x = u"? éí?§Dì?? ? ? ? ? ? ’"
    print x
    print x.encode('ascii', 'latin2ascii')

Why I return error.start + 1:

error object returned can be multiple characters, and we convert only first of these e.g. if I add print error.start, error.end to error handler output is

? éí?§Dì?? ? ? ? ? ? ’
0 1
2 10
3 10
4 10
5 10
6 10
7 10
8 10
9 10
11 12
13 14
15 16
17 18
19 20
21 22
1/4 einSDIeN >> 1/4 o R c '

so in second line we get chars from 2-10 but we convert only 2nd hence return 3 as continue point, if we return error.end output is

? éí?§Dì?? ? ? ? ? ? ’
0 1
2 10
11 12
13 14
15 16
17 18
19 20
21 22
1/4 e >> 1/4 o R c '

As we can see 2-10 portion has been replaced by a single char. off-course it would be faster to just encode whole range in one go and return error.end, but for demonstration purpose I have kept it simple.

see http://docs.python.org/library/codecs.html#codecs.register_error for more details

Answer 1

So here are three approaches, more or less as given or suggested in other answers:

# -*- coding: utf-8 -*-
import codecs
import unicodedata

x = u"Wikipédia, le projet d’encyclopédie"

xtd = {ord(u'’'): u"'", ord(u'é'): u'e', }

def asciify(error):
    return xtd[ord(error.object[error.start])], error.end

codecs.register_error('asciify', asciify)

def ae():
  return x.encode('ascii', 'asciify')

def ud():
  return unicodedata.normalize('NFKD', x).encode('ASCII', 'ignore')

def tr():
  return x.translate(xtd)

if __name__ == '__main__':
  print 'or:', x
  print 'ae:', ae()
  print 'ud:', ud()
  print 'tr:', tr()

Run as main, this emits:

or: Wikipédia, le projet d’encyclopédie
ae: Wikipedia, le projet d'encyclopedie
ud: Wikipedia, le projet dencyclopedie
tr: Wikipedia, le projet d'encyclopedie

showing clearly that the unicodedata-based approach, while it does have the convenience of not needing a translation map xtd, can't translate all characters properly in an automated fashion (it works for accented letters but not for the reverse-apostrophe), so it would also need some auxiliary step to deal explicitly with those (no doubt before what's now its body).

Performance is also interesting. On my laptop with Mac OS X 10.5 and system Python 2.5, quite repeatably:

$ python -mtimeit -s'import a' 'a.ae()'
100000 loops, best of 3: 7.5 usec per loop
$ python -mtimeit -s'import a' 'a.ud()'
100000 loops, best of 3: 3.66 usec per loop
$ python -mtimeit -s'import a' 'a.tr()'
10000 loops, best of 3: 21.4 usec per loop

translate is surprisingly slow (relative to the other approaches). I believe the issue is that the dict is looked into for every character in the translate case (and most are not there), but only for those few characters that ARE there with the asciify approach.

So for completeness here's "beefed-up unicodedata" approach:

specstd = {ord(u'’'): u"'", }
def specials(error):
  return specstd.get(ord(error.object[error.start]), u''), error.end
codecs.register_error('specials', specials)

def bu():
  return unicodedata.normalize('NFKD', x).encode('ASCII', 'specials')

this gives the right output, BUT:

$ python -mtimeit -s'import a' 'a.bu()'
100000 loops, best of 3: 10.7 usec per loop

...speed isn't all that good any more. So, if speed matters, it's no doubt worth the trouble of making a complete xtd translation dict and using the asciify approach. When a few extra microseconds per translation are no big deal, one might want to consider the bu approach simply for its convenience (only needs a translation dict for, hopefully few, special characters that don't translate correctly with the underlying unicodedata idea).