apache - Python CGI os.system causing malformed header

Question

I am running Apache/2.4.10 (Raspbian) and I am using python for CGI. But when I try to use os.system in simple code I get this malformed header error:

[Wed Aug 31 17:10:05.715740 2016] [cgid:error] [pid 3103:tid 1929376816] [client 192.168.0.106:59277] malformed header from script'play.cgi': Bad header: code.cgi

Here is the code from play.cgi:

#!/usr/bin/python
# -*- coding: UTF-8 -*-
import cgi
import os

print('Content-type: text/html')
print('')

os.system('ls')

The strange thing is that if I remove the os.system line it mysteriously starts working again. I have tried using popen instead, same problem. I have tried to obscure it in some code,change file name, different encodings and even time.sleep, none of those worked.

The strangest thing is that it works perfectly fine in a more complicated code.

Answer 1

To see why the problem happened, try launching your script

python webls.py > output

And than open output with some text editor. You will notice that your Content-type: text/html ended up being in the bottom of the file, which, of course, is wrong.

That happens because incorporating outout of your os.system in output of Python code screws thing up (because think about it: your print(...) is accumulated and flushed in blocks when appropriate, while os.system() abruptively prints data and because it is os.system transacton, flushes only its result (which is also why you do not see the problem if outout is to console)). The solution is to flush output after printing headers. You should change your code to

#!/usr/bin/python

import cgi
import os
import sys

print 'Content-type: text/html'
print ''

sys.stdout.flush()

os.system('ls')

---

Although that is a fix, you know you are doing something terribly wrong if you need to incorporate output of your console command to the content of the web page and use os.system for that. There are several things you should consider doing. These solutions are sorted by how recommended they are (from crappy to good):

-Use redirection of input/output. Save output of ls to file and read it in your Python code:

os.system('ls > /tmp/lsoutput')
print open('/tmp/lsoutput', 'r').read()

-Use subprocess. It allows you to capture output of console program and use it in your python code (example from python getoutput() equivalent in subprocess)

import subprocess
process = subprocess.Popen(['ls', '-a'], stdout=subprocess.PIPE)
out, err = process.communicate()
print(out)

-Do not call external programs at all, this is a bad thing to do. If you need a list of files, use Python functions instead

import os

for filename in os.listdir('.'):
    print filename