We can't guess what you are trying to do, nor what's in your code, not what "setting many different codecs" means, nor what u"string" is supposed to do for you.
Please change your code to its initial state so that it reflects as best you can what you are trying to do, run it again, and then edit your question to provide (1) the full traceback and error message that you get (2) snippet encompassing the last statement in your script that appears in the traceback (3) a brief description of what you want the code to do (4) what version of Python you are running.
Edit after details added to question:
(0) Let's try some transformations on the failing statement:
Original:
print "Error reading file %s"%u"%s/%s"%(folder, f)
Add spaces for reduced illegibility:
print "Error reading file %s" % u"%s/%s" % (folder, f)
Add parentheses to emphasise evaluation order:
print ("Error reading file %s" % u"%s/%s") % (folder, f)
Evaluate the (constant) expression in parentheses:
print u"Error reading file %s/%s" % (folder, f)
Is that really what you intended? Suggestion: construct the path ONCE, using a better method (see point (2) below).
(1) In general, use repr(foo) or "%r" % foo for diagnostics. That way, your diagnostic code is much less likely to cause an exception (as is happening here) AND you avoid ambiguity. Insert the statement print repr(folder), repr(f) before you try to get the size, rerun, and report back.
(2) Don't make paths by u"%s/%s" % (folder, filename) ... use os.path.join(folder, filename)
(3) Don't have bare excepts, check for known problems. So that unknown problems don't remain unknown, do something like this:
try:
some_code()
except ReasonForBaleOutError:
continue
except:
# something's gone wrong, so get diagnostic info
print repr(interesting_datum_1), repr(interesting_datum_2)
# ... and get traceback and error message
raise
A more sophisticated way would involve logging instead of printing, but the above is much better than not knowing what's going on.
Further edits after rtm("os.walk"), remembering old legends, and re-reading your code:
(4) os.walk() walks over the whole tree; you don't need to call it recursively.
(5) If you pass a unicode string to os.walk(), the results (paths, filenames) are reported as unicode. You don't need all that u"blah" stuff. Then you just have to choose how you display the unicode results.
(6) Removing paths with "$" in them: You must modify the list in situ but your method is dangerous. Try something like this:
for i in xrange(len(folders), -1, -1):
if '$' in folders[i]:
del folders[i]
(7) Your refer to files by joining a folder name and a file name. You are using the ORIGINAL folder name; when you rip out the recursion, this won't work; you'll need to use the currently-discarded content[0] value reported by os.walk.
(8) You should find yourself using something very simple like:
for folder, subfolders, filenames in os.walk(unicoded_top_folder):
There's no need for generator = os.walk(...); try: content = generator.next() etc and BTW if you ever need to do generator.next() in the future, use except StopIteration instead of a bare except.
(9) If the caller provides a non-existent folder, no exception is raised, it just does nothing. If the provided folder is exists but is empty, ditto. If you need to distinguish between these two scenarios, you'll need to do extra testing yourself.
Response to this comment from the OP: """Thanks, please read the info repr() has shown in the first post. I don't know why it printed so many different items, but it looks like they all have problems. And the common thing between all of them is they are .ink files. May that be the problem? Also, in the last ones, the firefox ones, it prints ( Modalitrovvisoria) while the real file name from Explorer contains ( Modalità provvisoria)"""
(10) Umm that's not ".INK".lower(), it's ".LNK".lower() ... perhaps you need to change the font in whatever you're reading that with.
(11) The fact that the "problem" file names all end in ".lnk" /may/ be something to do with os.walk() and/or Windows doing something special with the names of those files.
(12) I repeat here the Python statement that you used to produce that output, with some whitespace introduced :
print repr(
"Error reading file %s"
% u"%s/%s" % (
folder.decode('utf-8','ignore'),
f.decode('utf-8','ignore')
)
)
It seems that you have not read, or not understood, or just ignored, the advice I gave you in a comment on another answer (and that answerer's reply): UTF-8 is NOT relevant in the context of file names in a Windows file system.
We are interested in exactly what folder and f refer to. You have trampled all over the evidence by attempting to decode it using UTF-8. You have compounded the obfuscation by using the "ignore" option. Had you used the "replace" option, you would have seen "( Modalitufffdrovvisoria)". The "ignore" option has no place in debugging.
In any case, the fact that some of the file names had some kind of error but appeared NOT to lose characters with the "ignore" option (or appeared NOT to be mangled) is suspicious.
Which part of """Insert the statement print repr(folder), repr(f) """ did you not understand? All that you need to do is something like this:
print "Some meaningful text" # "error reading file" isn't
print "folder:", repr(folder)
print "f:", repr(f)
(13) It also appears that you have introduced UTF-8 elsewhere in your code, judging by the traceback: self.exploreRec(("%s/%s"%(folder, f)).encode("utf-8"), treshold)
I would like to point out that you still do not know whether folder and f refer to str objects or unicode objects, and two answers have suggested that they are very likely to be str objects, so why introduce blahbah.encode() ??
A more general point: Try to understand what your problem(s) is/are, BEFORE changing your script. Thrashing about trying every suggestion coupled with near-zero effective debugging technique is not the way forward.
(14) When you run your script again, you might like to reduce the volume of the output by running it over some subset of C: ... especially if you proceed with my original suggestion to have debug printing of ALL file names, not just the erroneous ones (knowing what non-error ones look like could help in understanding the problem).
Response to Bryan McLemore's "clean up" function:
(15) Here is an annotated interactive session that illustrates what actually happens with os.walk() and non-ASCII file names:
C:junkerabytest>dir
[snip]
Directory of C:junkerabytest
20/11/2009 01:28 PM <DIR> .
20/11/2009 01:28 PM <DIR> ..
20/11/2009 11:48 AM <DIR> empty
20/11/2009 01:26 PM 11 Ha?ek.txt
20/11/2009 01:31 PM 1,419 tbyte1.py
29/12/2007 09:33 AM 9 D.txt
3 File(s) 1,439 bytes
[snip]
C:junkerabytest>python26python
Python 2.6.4 (r264:75708, Oct 26 2009, 08:23:19) [MSC v.1500 32 bit (Intel)] onwin32
Type "help", "copyright", "credits" or "license" for more information.
>>> from pprint import pprint as pp
>>> import os
os.walk(unicode_string) -> results in unicode objects
>>> pp(list(os.walk(ur"c:junkerabytest")))
[(u'c:\junk\terabytest',
[u'empty'],
[u'Hau0161ek.txt', u'tbyte1.py', u'xd0.txt']),
(u'c:\junk\terabytest\empty', [], [])]
os.walk(str_string) -> results in str objects
>>> pp(list(os.walk(r"c:junkerabytest")))
[('c:\junk\terabytest',
['empty'],
['Hax9aek.txt', 'tbyte1.py', 'xd0.txt']),
('c:\junk\terabytest\empty', [], [])]
cp1252 is the encoding I'd expect to be used on my system ...
>>> u'u0161'.encode('cp1252')
'x9a'
>>> 'Hax9aek'.decode('cp1252')
u'Hau0161ek'
decoding the str with UTF-8 doesn't work, as expected
>>> 'Hax9aek'.decode('utf8')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:python26libencodingsutf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0x9a in position 2: unexpected code byte
ANY random string of bytes can be decoded without error using latin1
>>> 'Hax9aek'.decode('latin1')
u'Hax9aek'
BUT U+009A is a control character (SINGLE CHARACTER INTRODUCER), i.e. meaningless gibberish; absolutely nothing to do with the correct answer
>>> unicodedata.name(u'u0161')
'LATIN SMALL LETTER S WITH CARON'
>>>
(16) That example shows what happens when the character is representable in the default character set; what happens if it's not? Here's an example (using IDLE this time) of a file name containing CJK ideographs, which definitely aren't representable in my default character set:
IDLE 2.6.4
>>> import os
>>> from pprint import pprint as pp
repr(Unicode results) looks fine
>>> pp(list(os.walk(ur"c:junkerabytestchinese")))
[(u'c:\junk\terabytest\chinese', [], [u'nihaou4f60u597d.txt'])]
and the unicode displays just fine in IDLE:
>>> print list(os.walk(ur"c:junkerabytestchinese"))[0][2][0]
nihao你好.txt
The str result is evidently produced by using .encode(whatever, "replace") -- not very useful e.g. you can't open the file by passing that as the file name.
>>> pp(list(os.walk(r"c:junkerabytestchinese")))
[('c:\junk\terabytest\chinese', [], ['nihao??.txt'])]
So the conclusion is that for best results, one should pass a unicode string to os.walk(), and deal with any display problems.
