python - Importing correctly with pytest

Question

I just got set up to use pytest with Python 2.6. It has worked well so far with the exception of handling "import" statements: I can't seem to get pytest to respond to imports in the same way that my program does.

My directory structure is as follows:

src/
    main.py
    util.py
    test/
        test_util.py
    geom/
        vector.py
        region.py
        test/
            test_vector.py
            test_region.py

To run, I call python main.py from src/.

In main.py, I import both vector and region with

from geom.region import Region
from geom.vector import Vector

In vector.py, I import region with

from geom.region import Region

These all work fine when I run the code in a standard run. However, when I call "py.test" from src/, it consistently exits with import errors.

---

Some Problems and My Solution Attempts

My first problem was that, when running "test/test_foo.py", py.test could not "import foo.py" directly. I solved this by using the "imp" tool. In "test_util.py":

import imp
util = imp.load_source("util", "util.py")

This works great for many files. It also seems to imply that when pytest is running "path/test/test_foo.py" to test "path/foo.py", it is based in the directory "path".

However, this fails for "test_vector.py". Pytest can find and import the vector module, but it cannot locate any of vector's imports. The following imports (from "vector.py") both fail when using pytest:

from geom.region import *
from region import *

These both give errors of the form

ImportError: No module named [geom.region / region]

I don't know what to do next to solve this problem; my understanding of imports in Python is limited.

What is the proper way to handle imports when using pytest?

---

Edit: Extremely Hacky Solution

In vector.py, I changed the import statement from

from geom.region import Region

to simply

from region import Region

This makes the import relative to the directory of "vector.py".

Next, in "test/test_vector.py", I add the directory of "vector.py" to the path as follows:

import sys, os
sys.path.append(os.path.realpath(os.path.dirname(__file__)+"/.."))

This enables Python to find "../region.py" from "geom/test/test_vector.py".

This works, but it seems extremely problematic because I am adding a ton of new directories to the path. What I'm looking for is either

1) An import strategy that is compatible with pytest, or

2) An option in pytest that makes it compatible with my import strategy

So I am leaving this question open for answers of these kinds.

Answer 1

The issue here is that Pytest walks the filesystem to discover files that contain tests, but then needs to generate a module name that will cause import to load that file. (Remember, files are not modules.)

Pytest comes up with this test package name by finding the first directory at or above the level of the file that does not include an __init__.py file and declaring that the "basedir" for the module tree containing a module generated from this file. It then adds the basedir to sys.path and imports using the module name that will find that file relative to the basedir.

There are some implications of this of which you should beware:

1. The basepath may not match your intended basepath in which case the module will have a name that doesn't match what you would normally use. E.g., what you think of as geom.test.test_vector will actually be named just test_vector during the Pytest run because it found no __init__.py in src/geom/test/ and so added that directory to sys.path.

2. You may run into module naming collisions if two files in different directories have the same name. For example, lacking __init__.py files anywhere, adding geom/test/test_util.py will conflict with test/test_util.py because both are loaded as import test_util.py, with both test/ and geom/test/ in the path.

The system you're using here, without explicit __init__.py modules, is having Python create implicit namespace packages for your directories. (A package is a module with submodules.) Ideally we'd configure Pytest with a path from which it would also generate this, but it doesn't seem to know how to do that.

The easiest solution here is simply to add empty __init__.py files to all of the subdirectories under src/; this will cause Pytest to import everything using package/module names that start with directory names under src/.

The question How do I Pytest a project using PEP 420 namespace packages? discusses other solutions to this.