Explain how Windows batch newline variable hack works

Question

Can someone please explain how this works?

@echo off
REM Creating a Newline variable (the two blank lines are required!)
set NLM=^


set NL=^^^%NLM%%NLM%^%NLM%%NLM%
REM Example Usage:
echo There should be a newline%NL%inserted here.

emits:

There should be a newline
inserted here.

from How can you echo a newline in batch files?

Answer 1

The trick uses the behaviour of the caret.
Also explained at Long commands split over multiple lines in Windows Vista batch (.bat) file

The caret is an escape character for the next character, or at the line end it is used as multiline character, but this is nearly the same.

At the line end it simply escapes the next character, in this case the <Linefeed>, but there is a hidden feature, so if the escaped character is a <LF> it is ignored and the next character is read and escaped, but this charater will be always escaped, even if it is also a <LF>.

Now you can understand

set NLM=^


rem Two empty lines are required here

The NLM-Variable contains exactly one <LF> character. But if you try to use it with echo Line1%NLM%Line2 it fails, as the parser stops parsing at a single <LF>.
But this works

echo Line1^

Line2

So you need to add an escaped linefeed into the line and that is the NL-Variable. The NL-Variable consists of only three characters.
NL=^<LF><LF> And if this is expanded, it creates only one escaped <LF> as the first <LF> after the caret will be ignored.

Btw. In my opinion, it is much easier to use linefeeds with delayed expansion, as there is no need to escape anything.
In this example I use %=EMPTY=% instead of an empty line (for self commenting), but as the variable =EMPTY= can't exists it will be expanded to an empty line.

setlocal EnableDelayedExpansion
(set NLM=^
%=EMPTY=%
)
echo Line1!NLM!Line2

EDIT: Append some hints for useful using the <LF>

1) Use it as newline in an echo

setlocal EnableDelayedExpansion
(set LF=^
%=EMPTY=%
)
echo Line1!LF!Line2

2) Use it to split commands in a parenthesis block

setlocal EnableDelayedExpansion
(set LF=^
%=EMPTY=%
)
(
    echo Line1%LF%set x=4%LF%echo !x!%LF%
)

3) Create a (nearly) empty EOL-chararcter in a FOR /F loop, as <LF> is the line delimiter an EOL of <LF> is the same than an empty one.

FOR /F ^"eol^=^

delims^=^" %%a in (myFile.php) do echo %%a

4) Use LF for splitting text in a FOR /F loop

setlocal EnableDelayedExpansion
(set LF=^
%=EMPTY=%
)
set "var=Content1;Content2"
FOR /F "delims=" %%a in ("%var:;=!LF!%") do (
  echo %%a
)