xml - What is the Unicode U+001A Character? Aka 0x1A

Question

The U+001A character appears frequently in error messages relating to character encoding. What is the U+001A character?

Answer 1

U+001A is defined in the Unicode Standard as a control character with the name SUBSTITUTE, and it belongs to a group characterized as follows, in chapter 16 of the standard: “There are 65 code points set aside in the Unicode Standard for compatibility with the C0 and C1 control codes defined in the ISO/IEC 2022 framework [...] The Unicode Standard provides for the intact interchange of these code points, neither adding to nor subtracting from their semantics. The semantics of the control codes are generally determined by the application with which they are used. However, in the absence of specific application uses, they may be interpreted according to the control function semantics specified in ISO/IEC 6429:1992.”

ISO 6429 is effectively equivalent to ECMA 48, which mentions this code as having the short name SUB, too, and defines it as follows: “SUB is used in the place of a character that has been found to be invalid or in error. SUB is intended to be introduced by automatic means.” This reflects the definition of this control code in Ascii.

Thus, in general, U+001A may be used to indicate a character-level data error, such as the presence of bytes, in purported character data, that have no interpretation in the character encoding being applied. Loosely speaking, it would thus mean “bad character data”, but more appropriately “malformed data, when trying to interpret data as characters”. However, in Unicode, U+FFFD REPLACEMENT CHARACTER is more appropriate, as it has specific Unicode semantics.

Since the question has been tagged with “xml”, it needs to be noted that in XML 1.0, U+001A is forbidden, by clause 2.2 Characters. Note that the comment “any Unicode character, excluding the surrogate blocks, FFFE, and FFFF” is misleading (but comments are non-normative); U+001A is a Unicode character, though it is not a graphic character and its effect is not defined in the Unicode Standard.