I started wondering what behavior these constants have when I saw these constants at the htmlspecialchars page. The documentation was rubbish, so I started digging in the source code of PHP.
Basically, these constants affect whether certain entities are encoded or not (or decoded for html_entity_decode). The most obvious effect is whether the apostrophe (') is encoded to ' (for ENT_HTML401) or ' (for others). Similarly, it determines whether ' is decoded or not when using html_entity_decode. (' is always decoded).
All usages can be found in ext/standard/html.c and its header file. From ext/standard/html.h:
#define ENT_HTML_DOC_HTML401 0
#define ENT_HTML_DOC_XML1 16
#define ENT_HTML_DOC_XHTML 32
#define ENT_HTML_DOC_HTML5 (16|32)
(replace ENT_HTML_DOC_ by ENT_ to get their PHP constant names)
I started looking for all occurrences of these constants, and can share the following on the behaviour of the ENT_* constants:
- It affects which numeric entities will be decoded or not. For example,
 gets decoded to an unreadable/invalid character for ENT_HTML401, and ENT_XHTML and ENT_XML1. For ENT_HTML5 however, this is considered an invalid character and hence it stays . (C function unicode_cp_is_allowed)
- With
ENT_SUBSTITUTE enabled, invalid code unit sequences for a specified character set are replaced with ?. (does not depend on document type!)
- With
ENT_DISALLOWED enabled, code points that are disallowed for the specified document type are replaced with ?. (does not depend on charset!)
- With
ENT_IGNORE, the same invalid code unit sequences from ENT_SUBSTITUTE are removed and no replacement is done (depends on choice of "document type", e.g. ENT_HTML5)
- Disallow

 for ENT_HTML5 (line 976)
ENT_XHTML shares the entity map with ENT_HTML401. The only difference is that ' will be converted to an apostrophe with ENT_XHTML while ENT_HTML401 does not convert it (see this line)ENT_HTML401 and ENT_XHTML use exactly the same entity map (minus the difference from the previous point). ENT_HTML5 uses its own map. Others (currently ENT_XML1) have a very limited decoding map (>, &, <, ', " and their numeric equivalents). (see C function unescape_inverse_map)- Note for the previous point: when only a few entities must be escaped (think of
htmlspecialchars), all entities map will use the same one as ENT_XML1, except for ENT_HTML401. That one will not use ', but '.
That covers almost everything. I am not going to list all entity differences, instead I would like to point at https://github.com/php/php-src/tree/php-5.4.11/ext/standard/html_tables for some text files that contain the mappings for each type.
What ENT_* should I use for htmlspecialchars?
When using htmlspecialchars with ENT_COMPAT (default) or ENT_NOQUOTES, it does not matter which one you pick (see below). I saw some answers here on SO that boils down to this:
<input value="<?php echo htmlspecialchars($str, ENT_HTML5);?>" >
This is insecure. It will override the default value ENT_HTML401 | ENT_COMPAT which has as difference that HTML5 entities are used, but also that quotes are not escaped anymore! In addition, this is redundant code. The entities that have to be encoded by htmlspecialchars are the same for all ENT_HTML401, ENT_HTML5, etc.
Just use ENT_COMPAT or ENT_QUOTES instead. The latter also works when you use apostrophes for attributes (value='foo'). If you only have two arguments for htmlspecialchars, do not include the argument at all since it is the default (ENT_HTML401 is 0, remember?).
When you want to print something on the page (between tags, not attributes), it does not matter at all which one you pick as it will have equal effect. It is even sufficient to use ENT_NOQUOTES | ENT_HTML401 which equals to the numeric value 0.
See also below, about ENT_SUBTITUTE and ENT_DISALLOWED.
What ENT_* should I use for htmlentities?
If your text editor or database is so crappy that you cannot include non-US-ASCII characters (e.g. UTF-8), you can use htmlentities. Otherwise, save some bytes and use htmlspecialchars instead (see above).
Whether you need to use ENT_HTML401, ENT_HTML5 or something else depends on how your page is served. When you have a HTML5 page (<!doctype html>), use ENT_HTML5. XHTML or XML? Use the corresponding ENT_XHTML or ENT_XML1. With no doctype or plain ol' HTML4, use ENT_HTML401 (which is the default when omitted).
Should I use ENT_DISALLOWED, ENT_IGNORE or ENT_SUBSTITUTE?
By default, byte sequences that are invalid for the given character set are removed. To have a ? in place of an invalid byte sequence, specify ENT_SUBSTITUTE. (note that &#FFFD; is shown for non-UTF-8 charsets). When you specify ENT_IGNORE though, these characters are not shown even if you specified ENT_SUBSTITUTE.
Invalid characters for a document type are substituted by the same replacement character (or its entity) above when ENT_DISALLOWED is specified. This happens regardless of having ENT_IGNORE set (which has nothing to do with invalid chars for doctypes).
