c - Function optimized to infinite loop at 'gcc -O2'

Question

Context
I was asked the following puzzle by one of my friends:

void fn(void)
{
  /* write something after this comment so that the program output is 10 */
  /* write something before this comment */
}

int main()
{
  int i = 5;
  fn();
  printf("%d
", i);
  return 0;
}

I know there can be multiple solutions, some involving macro and some assuming something about the implementation and violating C.

One particular solution I was interested in is to make certain assumptions about stack and write following code: (I understand it is undefined behavior, but may work as expected on many implementations)

void fn(void)
{
  /* write something after this comment so that the program output is 10 */
  int a[1] = {0};
  int j = 0;
  while(a[j] != 5) ++j;  /* Search stack until you find 5 */
  a[j] = 10;             /* Overwrite it with 10 */
  /* write something before this comment */
}

Problem
This program worked fine in MSVC and gcc without optimization. But when I compiled it with gcc -O2 flag or tried on ideone, it loops infinitely in function fn.

My Observation
When I compiled the file with gcc -S vs gcc -S -O2 and compared, it clearly shows gcc kept an infinite loop in function fn.

Question
I understand because the code invokes undefined behavior, one can not call it a bug. But why and how does compiler analyze the behavior and leave an infinite loop at O2?

---

Many people commented to know the behavior if some of the variables are changed to volatile. The result as expected is:

• If i or j is changed to volatile, program behavior remains same.
• If array a is made volatile, program does not suffer infinite loop.
• Moreover if I apply the following patch

-  int a[1] = {0};
+  int aa[1] = {0};
+  int *a = aa;

The program behavior remains same (infinite loop)

If I compile the code with gcc -O2 -fdump-tree-optimized, I get the following intermediate file:

;; Function fn (fn) (executed once)

Removing basic block 3
fn ()
{
<bb 2>:

<bb 3>:
  goto <bb 3>;

}



;; Function main (main) (executed once)

main ()
{
<bb 2>:
  fn ();

}
Invalid sum of incoming frequencies 0, should be 10000

This verifies the assertions made after the answers below.

Answer 1

This is undefined behavior so the compiler can really do anything at all, we can find a similar example in GCC pre-4.8 Breaks Broken SPEC 2006 Benchmarks, where gcc takes a loop with undefined behavior and optimizes it to:

L2:
    jmp .L2

The article says (emphasis mine):

Of course this is an infinite loop. Since SATD() unconditionally executes undefined behavior (it’s a type 3 function), any translation (or none at all) is perfectly acceptable behavior for a correct C compiler. The undefined behavior is accessing d[16] just before exiting the loop. In C99 it is legal to create a pointer to an element one position past the end of the array, but that pointer must not be dereferenced. Similarly, the array cell one element past the end of the array must not be accessed.

which if we examine your program with godbolt we see:

fn:
.L2:
    jmp .L2

The logic being used by the optimizer probably goes something like this:

• All the elements of a are initialized to zero
• a is never modified before or within the loop
• So a[j] != 5 is always true -> infinite loop
• Because of the infinite, the a[j] = 10; is unreachable and so that can be optimized away, so can a and j since they are no longer needed to determine the loop condition.

which is similar to the case in the article which given:

int d[16];

analyzes the following loop:

for (dd=d[k=0]; k<16; dd=d[++k]) 

like this:

upon seeing d[++k], is permitted to assume that the incremented value of k is within the array bounds, since otherwise undefined behavior occurs. For the code here, GCC can infer that k is in the range 0..15. A bit later, when GCC sees k<16, it says to itself: “Aha– that expression is always true, so we have an infinite loop.”

Perhaps an interesting secondary point, is whether an infinite loop is considered observable behavior(w.r.t. to the as-if rule) or not, which effects whether an infinite loop can also be optimized away. We can see from C Compilers Disprove Fermat’s Last Theorem that before C11 there was at least some room for interpretation:

Many knowledgeable people (including me) read this as saying that the termination behavior of a program must not be changed. Obviously some compiler writers disagree, or else don’t believe that it matters. The fact that reasonable people disagree on the interpretation would seem to indicate that the C standard is flawed.

C11 adds clarification to section 6.8.5 Iteration statements and is covered in more detail in this answer.