linux - how to detect invalid utf8 unicode/binary in a text file

Question

I need to detect corrupted text file where there are invalid (non-ASCII) utf-8, Unicode or binary characters.

???>t??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????w?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????o?????????????????????????????_????????????????????????????????????????????????????????????o???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????~??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????}???????????????????????????}w??????×???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????~????????????????????????????????????????????????????????????????????????????????????????????????????????????_??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????^?????????????????s???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????w??????????????????????????????????????????????????????????????????}????????????????????????????????????????????y????????????????????????????????????????????????????????????????????????????????????????????????????o???????????????????????????????????????????????????????????????????????????}??????

what I have tried:

iconv -f utf-8 -t utf-8 -c file.csv 

this converts a file from utf-8 encoding to utf-8 encoding and -c is for skipping invalid utf-8 characters. However at the end those illegal characters still got printed. Are there any other solutions in bash on linux or other languages?

Answer 1

Assuming you have your locale set to UTF-8 (see locale output), this works well to recognize invalid UTF-8 sequences:

grep -axv '.*' file.txt

Explanation (from grep man page):

• -a, --text: treats file as text, essential prevents grep to abort once finding an invalid byte sequence (not being utf8)
• -v, --invert-match: inverts the output showing lines not matched
• -x '.*' (--line-regexp): means to match a complete line consisting of any utf8 character.

Hence, there will be output, which is the lines containing the invalid not utf8 byte sequence containing lines (since inverted -v)