Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
275 views
in Technique[技术] by (71.8m points)

python - How to compile __init__.py file using cython on Windows

When I compile an arbitrary __init__.py file on Windows with setup.py build_ext --inplace command, it has an unresolvable external symbol error (i.e. "LINK : error LNK2001: An unresolvable external symbol PyInit___init__").

The local environment:

python3.7,
Cython 0.29.14,
window10 x64,
Microsoft Visual Studio 2017,

ctest/__init__.py

# cython: language_level=3
print('__init__')

setup.py

from distutils.core import setup
from Cython.Build import cythonize


def compile_code(name, filename):
    setup(
        name=name,
        ext_modules=cythonize(filename),
    )


if __name__ == '__main__':
    compile_code('a', 'ctest/__init__.py')

The information printed by the terminal:

Compiling ctest/__init__.py because it changed.
[1/1] Cythonizing ctest/__init__.py
running build_ext
building 'ctest.__init__' extension
creating build
creating buildemp.win-amd64-3.7
creating buildemp.win-amd64-3.7Release
creating buildemp.win-amd64-3.7Releasectest
C:Program Files (x86)Microsoft Visual Studio2017CommunityVCToolsMSVC14.16.27023inHostX86x64cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -Id:py37include -Id:py37incl
ude "-IC:Program Files (x86)Microsoft Visual Studio2017CommunityVCToolsMSVC14.16.27023include" "-IC:Program Files (x86)Windows KitsNETFXSDK4.6.1includeum" "-IC:Pro
gram Files (x86)Windows Kits10include10.0.18362.0ucrt" "-IC:Program Files (x86)Windows Kits10include10.0.18362.0shared" "-IC:Program Files (x86)Windows Kits10includ
e10.0.18362.0um" "-IC:Program Files (x86)Windows Kits10include10.0.18362.0winrt" "-IC:Program Files (x86)Windows Kits10include10.0.18362.0cppwinrt" /Tcctest/__init__
.c /Fobuildemp.win-amd64-3.7Releasectest/__init__.obj
__init__.c
C:Program Files (x86)Microsoft Visual Studio2017CommunityVCToolsMSVC14.16.27023inHostX86x64link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTU
AC:NO /LIBPATH:d:py37Libs /LIBPATH:D:ENVScpytrantestlibs /LIBPATH:D:ENVScpytrantestPCbuildamd64 "/LIBPATH:C:Program Files (x86)Microsoft Visual Studio2017CommunityVC
ToolsMSVC14.16.27023libx64" "/LIBPATH:C:Program Files (x86)Windows KitsNETFXSDK4.6.1libumx64" "/LIBPATH:C:Program Files (x86)Windows Kits10lib10.0.18362.0ucrtx6
4" "/LIBPATH:C:Program Files (x86)Windows Kits10lib10.0.18362.0umx64" /EXPORT:PyInit___init__ buildemp.win-amd64-3.7Releasectest/__init__.obj /OUT:C:Users76923Deskto
pcpythonrecordctest\__init__.cp37-win_amd64.pyd /IMPLIB:buildemp.win-amd64-3.7Releasectest\__init__.cp37-win_amd64.lib
LINK : error LNK2001: An unresolvable external symbol PyInit___init__
buildemp.win-amd64-3.7Releasectest\__init__.cp37-win_amd64.lib : fatal error LNK1120: An external command that cannot be parsed
error: command 'C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\bin\HostX86\x64\link.exe' failed with exit status 1120
See Question&Answers more detail:os

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Reply

0 votes
by (71.8m points)

Maybe this behaviour might be viewed as a small bug in distutils-package (as pointed out by @DavidW there is this open issue: https://bugs.python.org/issue35893). However, it also shows, that cythonizing/compiling __init__.py isn't very popular and uses some undocumented implementation details which might change in the future, so it could be wiser to refrain from meddling with __init__.py.

But if you must...


When a package is imported explicitly, e.g.

import ctest

or implicitly, e.g.

import ctest.something

The FileFinder will see that a package, and not a module, is imported and will try to load ctest/__init__.py instead of ctest.py (which most likely doesn't exists):

    # Check if the module is the name of a directory (and thus a package).
    if cache_module in cache:
        base_path = _path_join(self.path, tail_module)
        for suffix, loader_class in self._loaders:
            init_filename = '__init__' + suffix
            full_path = _path_join(base_path, init_filename)
            if _path_isfile(full_path):
                return self._get_spec(loader_class, fullname, full_path, [base_path], target)

Used suffix, loader_class are for loading __init__.so, __init__.py and __init__.pyc in this order (see also this SO-post). This means, __init__.so will be loaded instead of __init__.py if we manage to create one.

While __init__.py is executed, The property __name__ is the name of the package, i.e. ctest in your case, and not __init__ as one might think. Thus, the name of the init-function, Python-interpreter will call when loading the extension __init__.so is PyInit_ctest in your case (and not PyInit___init__ as one might think).

The above explains, why it all works on Linux out-of-the-box. What about Windows?

The loader can only use symbols from a so/dll which aren't hidden. Per default all symbols built with gcc are visible, but not for VisualStudio on Windows - where all symbols are hidden per default (see e.g. this SO-post).

However, the init-function of a C-extension must be visible (and only the init-function) so it can be called with help of the loader - the solution is to export this symbol (i.e. PyInit_ctest) while linking, in your case it is the wrong /EXPORT:PyInit___init__-option for the linker.

The problem can be found in distutils, or more precise in build_ext-class:

def get_export_symbols(self, ext):
    """Return the list of symbols that a shared extension has to
    export.  This either uses 'ext.export_symbols' or, if it's not
    provided, "PyInit_" + module_name.  Only relevant on Windows, where
    the .pyd file (DLL) must export the module "PyInit_" function.
    """
    initfunc_name = "PyInit_" + ext.name.split('.')[-1]
    if initfunc_name not in ext.export_symbols:
        ext.export_symbols.append(initfunc_name)
    return ext.export_symbols

Here, sadly ext.name has __init__ in it.

From here, one possible solution is easy : to override get_export_symbols, i.e. to add the following to your setup.py-file (read on for a even simpler version):

...
from distutils.command.build_ext import build_ext
def get_export_symbols_fixed(self, ext):
    names = ext.name.split('.')
    if names[-1] != "__init__":
        initfunc_name = "PyInit_" + names[-1]
    else:
        # take name of the package if it is an __init__-file
        initfunc_name = "PyInit_" + names[-2]
    if initfunc_name not in ext.export_symbols:
        ext.export_symbols.append(initfunc_name)
    return ext.export_symbols

# replace wrong version with the fixed:
build_ext.get_export_symbols = get_export_symbols_fixed
...

Calling python setup.py build_ext -i should be enough now (because __init__.so will be loaded rather than __init__.py).


However, as @DawidW has pointed out, Cython uses macro PyMODINIT_FUNC, which is defined as

#define PyMODINIT_FUNC Py_EXPORTED_SYMBOL PyObject*

with Py_EXPORTED_SYMBOL being marked as visible/exported on Windows:

#define Py_EXPORTED_SYMBOL __declspec(dllexport)

Thus, there is no need to mark the symbol as visible at the command line. Even worse, this is the reason for the warning LNK4197:

__init__.obj : warning LNK4197: export 'PyInit_ctest' specified multiple times; using first specification

as PyInit_test is marked as __declspec(dllexport) and exported via option /EXPORT: at the same time.

/EXPORT:-option will be skipped by distutils, if export_symbols is empty, we can use even a simpler version of command.build_ext:

...
from distutils.command.build_ext import build_ext
def get_export_symbols_fixed(self, ext):
    pass  # return [] also does the job!

# replace wrong version with the fixed:
build_ext.get_export_symbols = get_export_symbols_fixed
...

This is even better than the first version, as it also fixes warning LNK4197!


与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
OGeek|极客中国-欢迎来到极客的世界,一个免费开放的程序员编程交流平台!开放,进步,分享!让技术改变生活,让极客改变未来! Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...