python - Apache SetEnv not working as expected with mod_wsgi

Question

In a flask application I wrote, I make use of an external library which can be configured using environment variables. Note: I wrote this external library myself. So I could make changes if necessary. When running from the command line an running the flask server with:

# env = python virtual environment
ENV_VAR=foo ./env/bin/python myapp/webui.py

it all woks as expected. But after deploying it to apache, and using SetEnv it does not work anymore. In fact, printing out os.environ to stderr (so it shows up in the apache logs reveals, that the wsgi process seems to be in a very different environment (for one, os.environ['PWD'] seems to be way off. In fact, it points to my development folder.

To help identifying the problem, following are the relevant parts of the application as a standalone hello-world app. The error output, and observations are at the very end of the post.

App folder layout:

Python app:

.
├── myapp.ini
├── setup.py
└── testenv
    ├── __init__.py
    ├── model
    │?? └── __init__.py
    └── webui.py

Apache folder (/var/www/michel/testenv):

.
├── env
│?? ├── [...]
├── logs
│?? ├── access.log
│?? └── error.log
└── wsgi
└── app.wsgi

myapp.ini

[app]
somevar=somevalue

setup.py

from setuptools import setup, find_packages

setup(
    name="testenv",
    version='1.0dev1',
    description="A test app",
    long_description="Hello World!",
    author="Some Author",
    author_email="[email protected]",
    license="BSD",
    include_package_data=True,
    install_requires = [
      'flask',
      ],
    packages=find_packages(exclude=["tests.*", "tests"]),
    zip_safe=False,
)

testenv/init.py

# empty

testenv/model/init.py

from os.path import expanduser, join, exists
from os import getcwd, getenv, pathsep
import logging
import sys

__version__ = '1.0dev1'

LOG = logging.getLogger(__name__)

def find_config():
    """
    Searches for an appropriate config file. If found, return the filename, and
    the parsed search path
    """

    path = [getcwd(), expanduser('~/.mycompany/myapp'), '/etc/mycompany/myapp']
    env_path = getenv("MYAPP_PATH")
    config_filename = getenv("MYAPP_CONFIG", "myapp.ini")
    if env_path:
        path = env_path.split(pathsep)

    detected_conf = None
    for dir in path:
        conf_name = join(dir, config_filename)
        if exists(conf_name):
            detected_conf = conf_name
            break
    return detected_conf, path

def load_config():
    """
    Load the config file.
    Raises an OSError if no file was found.
    """
    from ConfigParser import SafeConfigParser

    conf, path = find_config()
    if not conf:
        raise OSError("No config file found! Search path was %r" % path)

    parser = SafeConfigParser()
    parser.read(conf)
    LOG.info("Loaded settings from %r" % conf)
    return parser

try:
    CONF = load_config()
except OSError, ex:
    # Give a helpful message instead of a scary stack-trace
    print >>sys.stderr, str(ex)
    sys.exit(1)

testenv/webui.py

from testenv.model import CONF

from flask import Flask

app = Flask(__name__)

@app.route('/')
def index():
    return "Hello World %s!" % CONF.get('app', 'somevar')

if __name__ == '__main__':
    app.debue = True
    app.run()

Apache config

<VirtualHost *:80>
    ServerName testenv-test.my.fq.dn
    ServerAlias testenv-test

    WSGIDaemonProcess testenv user=michel threads=5
    WSGIScriptAlias / /var/www/michel/testenv/wsgi/app.wsgi
    SetEnv MYAPP_PATH /var/www/michel/testenv/config

    <Directory /var/www/michel/testenv/wsgi>
        WSGIProcessGroup testenv
        WSGIApplicationGroup %{GLOBAL}
        Order deny,allow
        Allow from all
    </Directory>

    ErrorLog /var/www/michel/testenv/logs/error.log
    LogLevel warn

    CustomLog /var/www/michel/testenv/logs/access.log combined

</VirtualHost>

app.wsgi

activate_this = '/var/www/michel/testenv/env/bin/activate_this.py'
execfile(activate_this, dict(__file__=activate_this))

from os import getcwd
import logging, sys

from testenv.webui import app as application

# You may want to change this if you are using another logging setup
logging.basicConfig(stream=sys.stderr, level=logging.DEBUG)

LOG = logging.getLogger(__name__)
LOG.debug('Current path: {0}'.format(getcwd()))

# Application config
application.debug = False

# vim: set ft=python :

Error and observations

This is the output of the apache error log.

[Thu Jan 26 10:48:15 2012] [error] No config file found! Search path was ['/home/users/michel', '/home/users/michel/.mycompany/myapp', '/etc/mycompany/myapp']
[Thu Jan 26 10:48:15 2012] [error] [client 10.115.192.101] mod_wsgi (pid=17946): Target WSGI script '/var/www/michel/testenv/wsgi/app.wsgi' cannot be loaded as Python module.
[Thu Jan 26 10:48:15 2012] [error] [client 10.115.192.101] mod_wsgi (pid=17946): SystemExit exception raised by WSGI script '/var/www/michel/testenv/wsgi/app.wsgi' ignored.
[Thu Jan 26 10:48:15 2012] [error] [client 10.115.192.101] Traceback (most recent call last):
[Thu Jan 26 10:48:15 2012] [error] [client 10.115.192.101]   File "/var/www/michel/testenv/wsgi/app.wsgi", line 10, in <module>
[Thu Jan 26 10:48:15 2012] [error] [client 10.115.192.101]     from testenv.webui import app as application
[Thu Jan 26 10:48:15 2012] [error] [client 10.115.192.101]   File "/var/www/michel/testenv/env/lib/python2.6/site-packages/testenv-1.0dev1-py2.6.egg/testenv/webui.py", line 1, in <module>
[Thu Jan 26 10:48:15 2012] [error] [client 10.115.192.101]     from testenv.model import CONF
[Thu Jan 26 10:48:15 2012] [error] [client 10.115.192.101]   File "/var/www/michel/testenv/env/lib/python2.6/site-packages/testenv-1.0dev1-py2.6.egg/testenv/model/__init__.py", line 51, in <module>
[Thu Jan 26 10:48:15 2012] [error] [client 10.115.192.101]     sys.exit(1)
[Thu Jan 26 10:48:15 2012] [error] [client 10.115.192.101] SystemExit: 1

My first observation is that the environment variable MYAPP_PATH does not appear in os.environ (this is not visible in this output, but I tested it, and it's not there!). As such, the config "resolver" falls back to the default path.

And my second observation is the search path for the config file lists /home/users/michel as return value of os.getcwd(). I was actually expecting something inside /var/www/michel/testenv.

My instinct tells me, that the way I am doing config resolution is not right. Mainly because code is executed at import-time. This leads me to the idea, that maybe the config-resolution code is executed before the WSGI environment is properly set up. Am I onto something there?

Short discussion / tangential question

How would you do the config resolution in this case? Given that the "model" sub-folder is in reality an external module, which should also work in non-wsgi applications, and should provide a method to configure a database connection.

Personally, I like the way I search for config files while still being able to override it. Only, the fact that code is executed at import-time is making my spider-senses tingling like crazy. The rationale behind this: Configuration handling is completely hidden (abstraction-barrier) by fellow developers using this module, and it "just works". They just need to import the module (with an existing config-file of course) and can jump right in without knowing any DB details. This also gives them an easy way to work with different databases (dev/test/deployment) and switch between them easily.

Now, inside mod_wsgi it does not anymore :(

Update:

Just now, to test my above idea, I changed the webui.py to the following:

import os

from flask import Flask, jsonify

app = Flask(__name__)

@app.route('/')
def index():
    return jsonify(os.environ)

if __name__ == '__main__':
    app.debue = True
    app.run()

The output on the webpage is the following:

{
    LANG: "C",
    APACHE_RUN_USER: "www-data",
    APACHE_PID_FILE: "/var/run/apache2.pid",
    PWD: "/home/users/michel/tmp/testenv",
    APACHE_RUN_GROUP: "www-data",
    PATH: "/usr/local/bin:/usr/bin:/bin",
    HOME: "/home/users/michel/"
}

This shows the same environment as the one seen by other debugging methods. So my initial though was wrong. But now I realised something stranger yet. os.environment['PWD'] is set to the folder where I have my development files. This is not at all where the application is running. Stranger yet, os.getcwd() returns /home/users/michel? This is inconsistent with what I see in os.environ. Should it not be the same as os.environ['PWD']?

The most important problem remains though: Why is the value set by apache's SetEnv (MYAPP_PATH in this case) not found in os.environ?

Answer 1

Note that the WSGI environment is passed upon each request to the application in the environ argument of the application object. This environment is totally unrelated to the process environment which is kept in os.environ. The SetEnv directive has no effect on os.environ and there is no way through Apache configuration directives to affect what is in the process environment.

So you have to do something else other than getenviron or os.environ['PWD'] to get the MY_PATH from apache.

Flask adds the wsgi environ to the request, not app.environ, it is done by the underlaying werkzeug. So on each request to the application, apache will add the MYAPP_CONF key and youll access it wherever you can access request it seems as, request.environ.get('MYAPP_CONFIG') for example.