c - Use generated header file from Cython

Question

According to the documentation it is possible to use C header files generated from Cython. I have followed the Hello World example with no problem and now I want to try something different. I want to use a public declaration to be able to use a custom method. My code structure is the following:

• hello.pyx
• setup.py
• main.c

hello.pyx

cdef public void say_hello():
    print("Hello World")

setup.py

from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext

ext_modules = [
    Extension("hello", ["hello.pyx", "main.c"]),
]

setup(
  name='Hello app',
  cmdclass={'build_ext': build_ext},
  ext_modules=ext_modules
)

main.c

#include "hello.h"

int main(void){
    say_hello();
}

The main.c acts as a test file to verify that the say_hello() method works as intended. Building the setup file python3 setup.py build_ext produces the following output.

    running build_ext
    skipping 'hello.c' Cython extension (up-to-date)
    building 'hello' extension
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.5m -c hello.c -o build/temp.linux-x86_64-3.5/hello.o
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.5m -c main.c -o build/temp.linux-x86_64-3.5/main.o
    In file included from main.c:1:0:
    hello.h:26:1: error: unknown type name ‘PyMODINIT_FUNC’
     PyMODINIT_FUNC inithello(void);
     ^
    error:

 command 'x86_64-linux-gnu-gcc' failed with exit status 1

The hello.h file contains the following

    /* Generated by Cython 0.25.2 */

#ifndef __PYX_HAVE__hello
#define __PYX_HAVE__hello


#ifndef __PYX_HAVE_API__hello

#ifndef __PYX_EXTERN_C
  #ifdef __cplusplus
    #define __PYX_EXTERN_C extern "C"
  #else
    #define __PYX_EXTERN_C extern
  #endif
#endif

#ifndef DL_IMPORT
  #define DL_IMPORT(_T) _T
#endif

__PYX_EXTERN_C DL_IMPORT(void) say_hello(void);

#endif /* !__PYX_HAVE_API__hello */

#if PY_MAJOR_VERSION < 3
PyMODINIT_FUNC inithello(void);  // <-- Line 26
#else
PyMODINIT_FUNC PyInit_hello(void);
#endif

#endif /* !__PYX_HAVE__hello */

To my understanding, it seems that gcc in not able to get the right version of Python (I'm using Python 3.5). Is there a way to set that somehow? Also, if that is indeed the case, why isn't this linking taken care of when I run the python3 setup.py build_ext command?

I don't have much experience with C so I might be missing something.

Answer 1

I think the problem is your misconception, that distutils will build the executable for you. That is not the case.

Out goal is to use some python functionality from a C-program. Let's do the necessary steps by ourselves:

1. Using cython to produce hello.h and hello.c from the given pyx-file.
2. Using the created hello.h for importing the python functionality in a C-program (main.c).
3. Using a compiler to compile both files hello.c and main.c.
4. Using a linker to link the in the last step created object files to an executable.

The first step is easy:

#hello.pyx:
cdef public void say_hello():
    print("Hello World")


>>>python -m cython hello.pyx

Now we have hello.c and hello.h in our working directory. Your main.c is wrong and should look like the following:

//main.c
#include <Python.h> //needed
#include "hello.h"

int main(void){
    Py_Initialize();  //Needed!
    inithello();      //Needed! called PyInit_hello() for Python3
    say_hello();
    Py_Finalize();    //Needed!
} 

Important: If you use Python>=3.5 and Cython>=0.29 you must take multi-phase module initialisation into account, as described in this SO-post - otherwise the resulting program will crash.

I'm not sure why the result of the cython doesn't include Python.h (this would make it self-contained), but this is the way it is - you need to include it prior to hello.h. Also Py_Initialize() and Py_Finalize() should be called before and after the usage of the functionality from hello.h.Also you need to initialize the module hello with inithello() otherwise a segmentation fault will haunt you at the start of the executable.

Now we need to find the flags for compiling. This can be done with the utility /usr/bin/python-config (I use python2.7, you need to do the same for python3) and option --cflags:

>>> /usr/bin/python-config --cflags
-I/usr/include/python2.7 -I/usr/include/x86_64-linux-gnu/python2.7
-fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong
-Wformat -Werror=format-security  -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes

So let's build it:

>>>gcc -c hello.c -o hello.o <our cflags>
>>>gcc -c main.c -o main.o <our cflags>

Now we have both object files hello.o and main.o in our working directory. We need to link them and in order to find the right flags we use once again the python-config utility but this time with the --ldflags-option:

>>> /usr/bin/python-config --ldflags
--L/usr/lib/python2.7/config-x86_64-linux-gnu -L/usr/lib -lpython2.7 
-lpthread -ldl  -lutil -lm  -Xlinker -export-dynamic -Wl,-O1 -Wl,-Bsymbolic-functions

That means:

>>> gcc main.o hello.o -o prog <our ldflags>

And now we have finally our executable!

---

Actually, it is not exactly what is asked for, but there is another possibility to generate an executable from the cython code, by using the option --embed. With this switch on not only the module is cythonized but also a main-function is created. For this your hello.pyx could look as follows:

#hello.pyx:
cdef public void say_hello():
    print("Hello World")
##main:
say_hello()

It is possible also to use __name__=="__main__" trick, but not needed.

Now after running:

>>>python -m cython hello.pyx --embed

a main function is created in the resulting hello.c which takes care of setting up/initializing the python-environment. So we just can build and link it:

>>> gcc hello.c -o prog <our cflags> <our ldflags>

And we are done - no need to know, how to initialize the whole python-stuff right!