python - How to understand the equal sign '=' symbol in IMAP email text?

Question

I am currently using Python imaplib to process email text.

I use fetch command to fetch the raw data email from GMail server. However, I found one thing really tricky - the equal sign '='. It is not a normal equal sign but a special symbol.

For example:

1. '=' sometimes acts as the hyphenation mark at the end of text line:

   Depending upon your module selections, course lecturers may also contact yo=
   u with preparatory work over the next few weeks. It would be wise to start =
   reviewing the preparatory reading lists provided on the module syllabi now =
   

2. Sometimes, it acts as a escape mark similar to '%', for example:

   a=20b is actually a<SPACE>b
=46rom here is actually From here

I am totally confused about such weird notation. I think there must be a guidance to handle this because GMail can handle such thing correctly in their apps.

I see that this is related to HTML encoding, just like '%' will be encoded. But the problem is, all I get from the IMAP response is a string that contain this '=' symbol. How should I handle this? Using regular expression?

Answer 1

In a nutshell, an equal sign at the end of a line indicates a soft line break. An equal sign followed by two hexadecimal characters (0-9, A-F) encodes a single octet (byte).

This encoding scheme is called "quoted printable" and is defined in section 6.7 of RFC 2045. See items (1) and (5), in particular.

6.7.  Quoted-Printable Content-Transfer-Encoding

   The Quoted-Printable encoding is intended to represent data that
   largely consists of octets that correspond to printable characters in
   the US-ASCII character set.  It encodes the data in such a way that
   the resulting octets are unlikely to be modified by mail transport.
   If the data being encoded are mostly US-ASCII text, the encoded form
   of the data remains largely recognizable by humans.  A body which is
   entirely US-ASCII may also be encoded in Quoted-Printable to ensure
   the integrity of the data should the message pass through a
   character-translating, and/or line-wrapping gateway.

   In this encoding, octets are to be represented as determined by the
   following rules:

    (1)   (General 8bit representation) Any octet, except a CR or
          LF that is part of a CRLF line break of the canonical
          (standard) form of the data being encoded, may be
          represented by an "=" followed by a two digit
          hexadecimal representation of the octet's value.  The
          digits of the hexadecimal alphabet, for this purpose,
          are "0123456789ABCDEF".  Uppercase letters must be
          used; lowercase letters are not allowed.  Thus, for
          example, the decimal value 12 (US-ASCII form feed) can
          be represented by "=0C", and the decimal value 61 (US-
          ASCII EQUAL SIGN) can be represented by "=3D".  This
          rule must be followed except when the following rules
          allow an alternative encoding.

    (2)   (Literal representation) Octets with decimal values of
          33 through 60 inclusive, and 62 through 126, inclusive,
          MAY be represented as the US-ASCII characters which
          correspond to those octets (EXCLAMATION POINT through
          LESS THAN, and GREATER THAN through TILDE,
          respectively).

    (3)   (White Space) Octets with values of 9 and 32 MAY be
          represented as US-ASCII TAB (HT) and SPACE characters,

          respectively, but MUST NOT be so represented at the end
          of an encoded line.  Any TAB (HT) or SPACE characters
          on an encoded line MUST thus be followed on that line
          by a printable character.  In particular, an "=" at the
          end of an encoded line, indicating a soft line break
          (see rule #5) may follow one or more TAB (HT) or SPACE
          characters.  It follows that an octet with decimal
          value 9 or 32 appearing at the end of an encoded line
          must be represented according to Rule #1.  This rule is
          necessary because some MTAs (Message Transport Agents,
          programs which transport messages from one user to
          another, or perform a portion of such transfers) are
          known to pad lines of text with SPACEs, and others are
          known to remove "white space" characters from the end
          of a line.  Therefore, when decoding a Quoted-Printable
          body, any trailing white space on a line must be
          deleted, as it will necessarily have been added by
          intermediate transport agents.

    (4)   (Line Breaks) A line break in a text body, represented
          as a CRLF sequence in the text canonical form, must be
          represented by a (RFC 822) line break, which is also a
          CRLF sequence, in the Quoted-Printable encoding.  Since
          the canonical representation of media types other than
          text do not generally include the representation of
          line breaks as CRLF sequences, no hard line breaks
          (i.e. line breaks that are intended to be meaningful
          and to be displayed to the user) can occur in the
          quoted-printable encoding of such types.  Sequences
          like "=0D", "=0A", "=0A=0D" and "=0D=0A" will routinely
          appear in non-text data represented in quoted-
          printable, of course.

          Note that many implementations may elect to encode the
          local representation of various content types directly
          rather than converting to canonical form first,
          encoding, and then converting back to local
          representation.  In particular, this may apply to plain
          text material on systems that use newline conventions
          other than a CRLF terminator sequence.  Such an
          implementation optimization is permissible, but only
          when the combined canonicalization-encoding step is
          equivalent to performing the three steps separately.

    (5)   (Soft Line Breaks) The Quoted-Printable encoding
          REQUIRES that encoded lines be no more than 76
          characters long.  If longer lines are to be encoded
          with the Quoted-Printable encoding, "soft" line breaks
          must be used.  An equal sign as the last character on a
          encoded line indicates such a non-significant ("soft")
          line break in the encoded text.