python - What files are required for Py_Initialize to run?

Question

I am working on a simple piece of code that runs a Python function from a C/C++ application. In order to do this I set the PYTHONPATH and run initialize as follows:

Py_SetPythonHome("../Python27");
Py_InitializeEx(0);

Then I import my module and run my function. It works great.

I am now attempting to build an installer for my colleagues to run my code. I want to minimize the number of files I need to include in this installer, for obvious reasons.

Googling around the subject tells me that I should be able to include the files "Python27.lib" and "Python27.dll", then zip the "DLLs" and "Lib" folders and include them. However, when I attempt this, Py_Initialize fails.

A quick examination of what is causing this failure shows that Py_Initialize appears to depend upon a number of .pyc files in the Lib folder including (but not limited to warnings.pyc, _abcoll.pyc, _future_.pyc and the contents of the "encodings" folder.

I cannot understand why this would be. Any advice?

Answer 1

At the beginning, I wanted to say that there's no module required (at least no non-builtin one) for Py_InitializeEx, so the only requirement was python27.dll (BTW: python27.lib is not required, unless your colleagues want to link something against it - but that wouldn't be very easy w / o Python's Include dir).
I had this code (using Python 2.7.10 that I built using VStudio 10 (2010)):

#include <stdio.h>
#include <conio.h>
#include <Python.h>

int main()
{
    int i = 0;
    char *pyCode = 
        "s = "abc"
"
        "print s, 1234";
    Py_InitializeEx(0);
    i = PyRun_SimpleString(pyCode);
    Py_Finalize();
    printf("PyRun_SimpleString returned: %d
Press a key to exit...
", i);
    _getch();
    return 0;
}

It built fine, it ran OK from VStudio, and from the command line (after copying the .dll in its folder). But then I copied the .exe and .dll to another computer and when running, bang!!!

ImportError: No module named site

Considering that:

• I have no PYTHON* env vars set in neither of the consoles on the 2 machines where I ran the .exe (with different results)
• On both machines the Python installation is on the same path (I previously (years ago) modified it on the machine that doesn't work)

I don't know why it doesn't behave the same (one thing that I haven't check is that there might be some registry key on the machine that works?).

Note: site is a (.py(c)) module located under %PYTHON_INSTALL_DIR%Lib.

Then, I browsed Python's source code and I came across [GitHub]: python/cpython - (v2.7.10) cpython/Python/pythonrun.c (function Py_InitializeEx - currently line #141) - this is how I'm going to refer a point in the source code):

if (!Py_NoSiteFlag)
    initsite(); /* Module site */

while in initsite:

m = PyImport_ImportModule("site");

which is pretty obvious (Py_NoSiteFlag is 0).

Then I noticed that Py_NoSiteFlag is declared as extern __declspec(dllexport) ([MS.Docs]: Using extern to Specify Linkage, [MS.Docs]: dllexport, dllimport), so I modified my code to:

#include <stdio.h>
#include <conio.h>
#include <Python.h>

extern __declspec(dllimport) int Py_NoSiteFlag;


int main()
{
    int i = 0;
    char *pyCode = 
        "s = "abc"
"
        "print s, 1234";
    Py_NoSiteFlag = 1;
    Py_InitializeEx(0);
    i = PyRun_SimpleString(pyCode);
    Py_Finalize();
    printf("PyRun_SimpleString returned: %d
Press a key to exit...
", i);
    _getch();
    return 0;
}

and it works! Yay!

So, at this point only the .dll is required in order to run a piece of code. But I imagine that your code is "a little bit" more complex than that (it has imports ([Python 2.Docs]: The import statement). To solve the import problem, you can use this nice module: [Python 2.Docs]: modulefinder - Find modules used by a script (part of Python 2.7's standard modules). To make use of it:

• Save the code that you execute from C in a .py file
• Run modulefinder against it

Here's an example for my code (pyCode contents in my C program, saved in a file).

code00.py:

s = "abc"
print s, 1234

Running:

${PYTHON_INSTALL_DIR}python.exe -m modulefinder code00.py

yields:

Name                      File
----                      ----
m __main__                code00.py

But, if I add an import os (which is a pretty common module) statement in the file, the above command yields:

Name                        File
----                        ----
m StringIO                  %PYTHON_INSTALL_DIR%libStringIO.py
m UserDict                  %PYTHON_INSTALL_DIR%libUserDict.py
m __builtin__
m __future__                %PYTHON_INSTALL_DIR%lib\__future__.py
m __main__                  a.py
m _abcoll                   %PYTHON_INSTALL_DIR%lib\_abcoll.py
m _codecs
m _collections
m _functools
m _hashlib                  %PYTHON_INSTALL_DIR%DLLs\_hashlib.pyd
m _heapq
m _io
m _locale
m _random
m _sre
m _struct
m _subprocess
m _threading_local          %PYTHON_INSTALL_DIR%lib\_threading_local.py
m _warnings
m _weakref
m _weakrefset               %PYTHON_INSTALL_DIR%lib\_weakrefset.py
m abc                       %PYTHON_INSTALL_DIR%libabc.py
m array
m atexit                    %PYTHON_INSTALL_DIR%libatexit.py
m bdb                       %PYTHON_INSTALL_DIR%libdb.py
m binascii
m cPickle
m cStringIO
m cmd                       %PYTHON_INSTALL_DIR%libcmd.py
m codecs                    %PYTHON_INSTALL_DIR%libcodecs.py
m collections               %PYTHON_INSTALL_DIR%libcollections.py
m copy                      %PYTHON_INSTALL_DIR%libcopy.py
m copy_reg                  %PYTHON_INSTALL_DIR%libcopy_reg.py
m difflib                   %PYTHON_INSTALL_DIR%libdifflib.py
m dis                       %PYTHON_INSTALL_DIR%libdis.py
m doctest                   %PYTHON_INSTALL_DIR%libdoctest.py
m dummy_thread              %PYTHON_INSTALL_DIR%libdummy_thread.py
P encodings                 %PYTHON_INSTALL_DIR%libencodings\__init__.py
m encodings.aliases         %PYTHON_INSTALL_DIR%libencodingsaliases.py
m errno
m exceptions
m fnmatch                   %PYTHON_INSTALL_DIR%libfnmatch.py
m functools                 %PYTHON_INSTALL_DIR%libfunctools.py
m gc
m genericpath               %PYTHON_INSTALL_DIR%libgenericpath.py
m getopt                    %PYTHON_INSTALL_DIR%libgetopt.py
m gettext                   %PYTHON_INSTALL_DIR%libgettext.py
m hashlib                   %PYTHON_INSTALL_DIR%libhashlib.py
m heapq                     %PYTHON_INSTALL_DIR%libheapq.py
m imp
m inspect                   %PYTHON_INSTALL_DIR%libinspect.py
m io                        %PYTHON_INSTALL_DIR%libio.py
m itertools
m keyword                   %PYTHON_INSTALL_DIR%libkeyword.py
m linecache                 %PYTHON_INSTALL_DIR%liblinecache.py
m locale                    %PYTHON_INSTALL_DIR%liblocale.py
P logging                   %PYTHON_INSTALL_DIR%liblogging\__init__.py
m marshal
m math
m msvcrt
m nt
m ntpath                    %PYTHON_INSTALL_DIR%lib
tpath.py
m opcode                    %PYTHON_INSTALL_DIR%libopcode.py
m operator
m optparse                  %PYTHON_INSTALL_DIR%liboptparse.py
m os                        %PYTHON_INSTALL_DIR%libos.py
m os2emxpath                %PYTHON_INSTALL_DIR%libos2emxpath.py
m pdb                       %PYTHON_INSTALL_DIR%libpdb.py
m pickle                    %PYTHON_INSTALL_DIR%libpickle.py
m posixpath                 %PYTHON_INSTALL_DIR%libposixpath.py
m pprint                    %PYTHON_INSTALL_DIR%libpprint.py
m random                    %PYTHON_INSTALL_DIR%lib
andom.py
m re                        %PYTHON_INSTALL_DIR%lib
e.py
m repr                      %PYTHON_INSTALL_DIR%lib
epr.py
m select                    %PYTHON_INSTALL_DIR%DLLsselect.pyd
m shlex                     %PYTHON_INSTALL_DIR%libshlex.py
m signal
m sre_compile               %PYTHON_INSTALL_DIR%libsre_compile.py
m sre_constants             %PYTHON_INSTALL_DIR%libsre_constants.py
m sre_parse                 %PYTHON_INSTALL_DIR%libsre_parse.py
m stat                      %PYTHON_INSTALL_DIR%libstat.py
m string                    %PYTHON_INSTALL_DIR%libstring.py
m strop
m struct                    %PYTHON_INSTALL_DIR%libstruct.py
m subprocess                %PYTHON_INSTALL_DIR%libsubprocess.py
m sys
m tempfile                  %PYTHON_INSTALL_DIR%libempfile.py
m textwrap                  %PYTHON_INSTALL_DIR%libextwrap.py
m thread
m threading                 %PYTHON_INSTALL_DIR%libhreading.py
m time
m token                     %PYTHON_INSTALL_DIR%liboken.py
m tokenize                  %PYTHON_INSTALL_DIR%libokenize.py
m traceback                 %PYTHON_INSTALL_DIR%libraceback.py
m types                     %PYTHON_INSTALL_DIR%libypes.py
P unittest                  %PYTHON_INSTALL_DIR%libunittest\__init__.py
m unittest.case             %PYTHON_INSTALL_DIR%libunittestcase.py
m unittest.loader           %PYTHON_INSTALL_DIR%libunittestloader.py
m unittest.main             %PYTHON_INSTALL_DIR%libunittestmain.py
m unittest.result           %PYTHON_INSTALL_DIR%libunittest
esult.py
m unittest.runner           %PYTHON_INSTALL_DIR%libunittest
unner.py
m unittest.signals          %PYTHON_INSTALL_DIR%libunittestsignals.py
m unittest.suite            %PYTHON_INSTALL_DIR%libunittestsuite.py
m unittest.util             %PYTHON_INSTALL_DIR%libunittestutil.py
m warnings                  %PYTHON_INSTALL_DIR%libwarnings.py
m weakref                   %PYTHON_INSTALL_DIR%libweakref.py

 Missing modules:
? _emx_link imported from os
? ce imported from os
? fcntl imported from subprocess, tempfile
? org.python.core imported from copy, pickle
? os.path imported from os, shlex
? os2 imported from os
? posix imported from os
? pwd imported from posixpath
? readline imported from cmd, pdb
? riscos imported from os
? riscosenviron imported from os
? riscospath imported from os

As you can see, there is an awfully lot of modules (I modified the output a little bit, instead of the actual path I placed the ${PYTHON_INSTALL_DIR} env var placeholder). In order for the Python code to work, you'll have to include all of those modules/packages in the installer.

Notes about modulefinder's output (that I've noticed while playing with it):

• It searches for modules recursively, so here is the whole module dependency tree
• It searches for import statements located in functions (so, not only the ones at module level)
• It doesn't search for dynamic imports (e.g. [Python 2.Docs]: __import__(name[, globals[, locals[, fromlist[, level]]]]))

So, looking at the modul