shell - Python Script not running in crontab calling pysaunter

Question

I have read multiple posts and many articles detailing that scipts in a cron job need to keep the environment variables necessary to run inside the script itself due to the opening of shells within cron. My situation is unique in that my path variables are all being set as discussed, which in turn will successfully call the pysaunter python egg using subprocess.call(), but it seems to break down from there. This causes the whole process to break in a cron job.

For clarity, here are the steps I refer to:

1) cronjob calls run_test.py -n foo
2) run_test.py sets the environment variables correctly 
(cur_shell_path=sys.path (converted to proper path string, not shown here)
 my_env= os.environ.copy()
 my_env["PATH"] = my_env["PATH"] + cur_shell_path)
3) run_test.py calls subprocess.call("pysaunter -m foo -v", env=my_env, shell=True)

The output of step 3 shows that it is finding the egg and successfully starts to load the necessary modules from pysaunter, but then it breaks when attempting to find a directory used to modify pysaunter. The error reads :

ImportError: no module named helpers

I have attempted adding this path to the environment multiple times but it never seems to find the directory that contains helpers.py. The command pysaunter -m foo -v works normally when called from an interactive shell.

I couldn't find much help on pysaunter, so I assume that too much pysaunter specifics would be unneccessary here. If however you know more about pysaunter, please let me know if you require more information. I am not sure what to share.

I have also read many posts discussing the ability to change the default behavior of a shell by editing the .profile/.bash_profile. I attempted to find a place that would make my path variables globally accessible, but I couldn't find anything. I am not sure how this is done, and it may fix my problem, so if you know anything about that please let me know.

Final note, this is running on Mac 10.7.5.

Answer 1

After much trial and error, and many, many stackoverflow.com articles and other tutorials online, and with the help of a Perl script I found that did something similar, I was able to figure out what needed to be done to get this to work.

Here are the steps to making sure everything is set up correctly:

1. Make sure that you have the variables you need in PYTHONPATH (found here and here, and for more info. go here) inside the .profile or .bash_profile for any shell you want to test your script in to make sure it works.

2. Edit your crontab to include the directories needed to run your script in a cron job (found here and here).

   a) Be sure to include the root directory in the PATH variable (.) as explained here. Basically, if you are running an executable with your command it needs to be able to find root or the directory where the executable is stored and also probably these: (/sbin:/bin:/usr/sbin:/usr/bin).

3. In your crontab file, create a cronjob that will change the current directory to the directory where you have successfully run the script before (e.g., /Users/user/Documents/foo).

   a) This will look like the following:

   * * * * cd /Users/user/Documents/foo; bar -l doSomething -v 
   

4. Since my problem dealt specifically with calling an executable, it is necessary to note that there are multiple ways of writing the Python script to run (though in the process of discovery I learned that this works for any call made using subprocess in cron).

   The first method looks like this:

   ... #some script calls
   my_env = os.environ.copy()
   my_env["PYTHONPATH"] = "{}:{}".format(os.environ["PATH"] ,"<path you want to include>")
   os.chdir("<path/to/desired/directory>")
   subprocess.Popen(<call_as_string>, env=my_env, shell=True)
   

   And the second looks like this:

   ... #some script calls
   os.environ["PYTHONPATH"] = "{}:{}".format(os.environ["PATH"] ,"<path you want to include>")
   os.chdir("<path/to/desired/directory>")
   subprocess.Popen(<call_as_list_of_arguments)
   

   Since the executable needed the path to the helpers directory included in the shell where it was called from, it was necessary to pass the environment variable to the executable as explained here. What I found, though, is that modifying the PATH variable in the environment didn't work for a cron job, but setting the PYTHONPATH did. I read here that the PATH variable is used by the shell to only look for executables, so for a new shell inside a cronjob you need to pass the shell a PYTHONPATH to look for new Python modules. (This is also explained in the Python docs.)

   The differences between the two different methods are explained in the subprocess documentation cited in the question, but a good tutorial on this module can be found here.