python - Why doesn't `readline` return for `subprocess.stdout`?

Question

I use Python subprocess to fork a C application and open a pipe to stdout.

app = subprocess.Popen(args, stdout=subprocess.PIPE)

The application writes several lines to stdout like so:

printf("Line 0
");
printf("Line 1
");
printf("Line 2
");

I try to read these lines in my Python script before the application exits:

line = app.stdout.readline()

However, readline blocks indefinitely without returning any content, even though I expect to read Line 0, Line 1, and Line 2, in three separate calls to readline. I notice that when the application finally exits, readline returns the expected contents. However, I want readline to return the expected contents as soon as they are passed to printf. What is happening?

question from:https://stackoverflow.com/questions/65622933/why-doesnt-readline-return-for-subprocess-stdout

Answer 1

According to this StackOverflow question, printf only flushes its output on a newline when the output device is interactive (e.g., a terminal). When the output device is non-interactive (e.g., a Python subprocess pipe, a file, etc.), stdout is fully buffered, so the output will only be flushed when the buffer is full, when fflush(stdout) is manually called, or when the buffer is otherwise forced to flush (e.g., on process exit).

Thus, readline did not return because the application's stdout was never flushed, so readline had nothing to read. Changing the application code to this fixes the issue:

printf("Line 0
");
printf("Line 1
");
printf("Line 2
");
fflush(stdout);  // New line

Others may wish to set the buffer mode of stdout with setvbuf rather than call fflush.

I was surprised by this because the common understanding of printf is that it flushes on a newline character, but this does not happen in all cases.