Maybe this behaviour might be viewed as a small bug in distutils
-package (as pointed out by @DavidW there is this open issue: https://bugs.python.org/issue35893). However, it also shows, that cythonizing/compiling __init__.py
isn't very popular and uses some undocumented implementation details which might change in the future, so it could be wiser to refrain from meddling with __init__.py
.
But if you must...
When a package is imported explicitly, e.g.
import ctest
or implicitly, e.g.
import ctest.something
The FileFinder
will see that a package, and not a module, is imported and will try to load ctest/__init__.py
instead of ctest.py
(which most likely doesn't exists):
# Check if the module is the name of a directory (and thus a package).
if cache_module in cache:
base_path = _path_join(self.path, tail_module)
for suffix, loader_class in self._loaders:
init_filename = '__init__' + suffix
full_path = _path_join(base_path, init_filename)
if _path_isfile(full_path):
return self._get_spec(loader_class, fullname, full_path, [base_path], target)
Used suffix, loader_class
are for loading __init__.so
, __init__.py
and __init__.pyc
in this order (see also this SO-post). This means, __init__.so
will be loaded instead of __init__.py
if we manage to create one.
While __init__.py
is executed, The property __name__
is the name of the package, i.e. ctest
in your case, and not __init__
as one might think. Thus, the name of the init-function, Python-interpreter will call when loading the extension __init__.so
is PyInit_ctest
in your case (and not PyInit___init__
as one might think).
The above explains, why it all works on Linux out-of-the-box. What about Windows?
The loader can only use symbols from a so/dll which aren't hidden. Per default all symbols built with gcc are visible, but not for VisualStudio on Windows - where all symbols are hidden per default (see e.g. this SO-post).
However, the init-function of a C-extension must be visible (and only the init-function) so it can be called with help of the loader - the solution is to export this symbol (i.e. PyInit_ctest
) while linking, in your case it is the wrong /EXPORT:PyInit___init__
-option for the linker.
The problem can be found in distutils, or more precise in build_ext
-class:
def get_export_symbols(self, ext):
"""Return the list of symbols that a shared extension has to
export. This either uses 'ext.export_symbols' or, if it's not
provided, "PyInit_" + module_name. Only relevant on Windows, where
the .pyd file (DLL) must export the module "PyInit_" function.
"""
initfunc_name = "PyInit_" + ext.name.split('.')[-1]
if initfunc_name not in ext.export_symbols:
ext.export_symbols.append(initfunc_name)
return ext.export_symbols
Here, sadly ext.name
has __init__
in it.
From here, one possible solution is easy : to override get_export_symbols
, i.e. to add the following to your setup.py
-file (read on for a even simpler version):
...
from distutils.command.build_ext import build_ext
def get_export_symbols_fixed(self, ext):
names = ext.name.split('.')
if names[-1] != "__init__":
initfunc_name = "PyInit_" + names[-1]
else:
# take name of the package if it is an __init__-file
initfunc_name = "PyInit_" + names[-2]
if initfunc_name not in ext.export_symbols:
ext.export_symbols.append(initfunc_name)
return ext.export_symbols
# replace wrong version with the fixed:
build_ext.get_export_symbols = get_export_symbols_fixed
...
Calling python setup.py build_ext -i
should be enough now (because __init__.so
will be loaded rather than __init__.py
).
However, as @DawidW has pointed out, Cython uses macro PyMODINIT_FUNC
, which is defined as
#define PyMODINIT_FUNC Py_EXPORTED_SYMBOL PyObject*
with Py_EXPORTED_SYMBOL
being marked as visible/exported on Windows:
#define Py_EXPORTED_SYMBOL __declspec(dllexport)
Thus, there is no need to mark the symbol as visible at the command line. Even worse, this is the reason for the warning LNK4197:
__init__.obj : warning LNK4197: export 'PyInit_ctest' specified multiple times;
using first specification
as PyInit_test
is marked as __declspec(dllexport)
and exported via option /EXPORT:
at the same time.
/EXPORT:
-option will be skipped by distutils, if export_symbols
is empty, we can use even a simpler version of command.build_ext
:
...
from distutils.command.build_ext import build_ext
def get_export_symbols_fixed(self, ext):
pass # return [] also does the job!
# replace wrong version with the fixed:
build_ext.get_export_symbols = get_export_symbols_fixed
...
This is even better than the first version, as it also fixes warning LNK4197!