Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
496 views
in Technique[技术] by (71.8m points)

python - Print unicode string to console OK but fails when redirect to a file. How to fix?

I have Python 2.7.1 on a Simplified-Chinese version of Windows XP, and I have a program like this(windows_prn_utf8.py):

#!/usr/bin/env python
# -*- coding: utf8 -*-

print unicode('xE7x94xB5', 'utf8')

If I run it on Windows CMD console, it output the right Chinese character '电' ; however, if I try to redirect the command output to a file. I got error.

D:Temp>windows_prn_utf8.py > 1.txt
Traceback (most recent call last):
  File "D:Tempwindows_prn_utf8.py", line 4, in <module>
    print unicode('xE7x94xB5', 'utf8')
UnicodeEncodeError: 'ascii' codec can't encode character u'u7535' in position 0: ordinal not in range(128)

I realize there is a missing link here. There should be a way to determine, in case 1.txt is generated, whether the unicode character in 1.txt should be encoded in utf-8 or codepage-936 or other encodings.

Then how to fix it? My preference is to have utf-8 encoding in 1.txt . Thank you.

enter image description here

See Question&Answers more detail:os

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Reply

0 votes
by (71.8m points)

Seems like this was solved, but I think a bit more detail will help explain this actual problem.

The 'utf8' in unicode('xE7x94xB5', 'utf8') is telling the interpreter how to decode the 3 bytes you're providing in the other argument in order to represent the character internally as a unicode object:

In [6]: uobj = unicode('xe7x94xb5','utf8')

In [7]: uobj
Out[7]: u'u7535'

Another example would be creating the same character from its utf-16 representation (which is what python displays by default and shown in the Out[7] line above):

In [8]: uobj = unicode('x35x75','utf16')

In [9]: uobj
Out[9]: u'u7535'

In your example after the object has been created it becomes an argument to print which tries to write it to standard out (console window, redirected to a file, etc). The complication is that print must re-encode that object into a byte stream before writing it. It looks like in your case the encoding it used by default was ACSII which cannot represent that character.

(If a console will try to display the characters, they will be re-decoded and replaced in the window with the corresponding font glyphs--this is why your output and the console both need to be 'speaking' the same encoding.)

From what I've seen cmd.exe in windows is pretty confusing when it comes to character encodings, but what I do on other OSes is explicitly encode the bytes before printing/writing them with the unicode object's encode function. This returns an encoded byte sequence stored in a str object:

In [10]: sobj = uobj.encode('utf8')

In [11]: type(sobj)
Out[11]: str

In [12]: sobj
Out[12]: 'xe7x94xb5'

In [13]: print sobj
电

Now that print is given a str instead of a unicode, it doesn't need to encode anything. In my case my terminal was decoding utf8, and its font contained that particular character, so it was displayed properly on my screen (and hopefully right now in your browser).


与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
OGeek|极客中国-欢迎来到极客的世界,一个免费开放的程序员编程交流平台!开放,进步,分享!让技术改变生活,让极客改变未来! Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...