python - Including a compiled module in module that is wrapped with f2py (Minimum working example)?

Question

I have tried, but am failing, to get a minimum working example. As I do not need to expose much of my fortran code to python, I don't need f2py to wrap large parts of it. Also, due to allocatable arrays being passed and derived types being used, I specifically want f2py to only wrap the interface module I created (in the following example 'main.f90'). But I am having problems to get the other modules which I compile separately to link to my main module.

The Code:

Note all source files are in a single directory.

I have created a fortran module I want to compile (libtest.f90):

module testmod
implicit none
contains

  subroutine testsub(arr)
    real, allocatable, intent(in) :: arr(:,:)
    print *, 'testsub executed'
  end subroutine testsub

end module testmod

and a fortran module that I want to wrap with f2py (main.f90):

module mainmod

use testmod
implicit none
contains

  subroutine mainsub
    real, allocatable :: arr(:,:)
    call testsub(arr)
  end subroutine main sub

end module mainmod

I use the following compile commands:

gfortran -c -fPIC libtest.f90

which generates 'libtest.o' and 'testmod.mod', and

f2py -c --fcompiler=gfortran -L. -I. -llibtest -m Main main.f90

Which gives me 'ld: library not found for -llibtest'.

I don't understand why this occurs, since it seems to work for others (F2PY doesn't find a module). If I take out the -llibtest, I of course get (in my python script):

Traceback (most recent call last):
  File "./script.py", line 7, in <module>
    import Main
ImportError: dlopen(/Users/gmueller/Workspace/Minimum_PySpin/Main.so, 2): Symbol not found: ___testmod_MOD_testsub
  Referenced from: /Users/gmueller/Workspace/Minimum_PySpin/Main.so
  Expected in: dynamic lookup

Edit: Note I am on OSX(10.9.5), in case that makes some difference (but it shouldn't, since I do not need to pass any -shared (linux) or -dynamiclib (osx) to gfortran, right?).

Here the full output from f2py:

f2py -c --fcompiler=gfortran -L. -I. -llibtest -m Main main.f90
Unknown vendor: "gfortran"
running build
running config_cc
unifing config_cc, config, build_clib, build_ext, build commands --compiler options
running config_fc
unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options
running build_src
build_src
building extension "Main" sources
f2py options: []
f2py:> /var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/src.macosx-10.4-x86_64-2.7/Mainmodule.c
creating /var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/src.macosx-10.4-x86_64-2.7
Reading fortran codes...
    Reading file 'main.f90' (format:free)
Post-processing...
    Block: Main
            Block: mainmod
In: :Main:main.f90:mainmod
get_useparameters: no module testmod info used by mainmod
                Block: mainsub
In: :Main:main.f90:mainmod:mainsub
get_useparameters: no module testmod info used by mainsub
Post-processing (stage 2)...
    Block: Main
        Block: unknown_interface
            Block: mainmod
                Block: mainsub
Building modules...
    Building module "Main"...
        Constructing F90 module support for "mainmod"...
            Constructing wrapper function "mainmod.mainsub"...
              mainsub()
    Wrote C/API module "Main" to file "/var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/src.macosx-10.4-x86_64-2.7/Mainmodule.c"
    Fortran 90 wrappers are saved to "/var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/src.macosx-10.4-x86_64-2.7/Main-f2pywrappers2.f90"
  adding '/var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/src.macosx-10.4-x86_64-2.7/fortranobject.c' to sources.
  adding '/var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/src.macosx-10.4-x86_64-2.7' to include_dirs.
copying /usr/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/f2py/src/fortranobject.c -> /var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/src.macosx-10.4-x86_64-2.7
copying /usr/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/f2py/src/fortranobject.h -> /var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/src.macosx-10.4-x86_64-2.7
  adding '/var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/src.macosx-10.4-x86_64-2.7/Main-f2pywrappers2.f90' to sources.
build_src: building npy-pkg config files
running build_ext
customize UnixCCompiler
customize UnixCCompiler using build_ext
customize Gnu95FCompiler
Found executable /usr/local/bin/gfortran
customize Gnu95FCompiler using build_ext
building 'Main' extension
compiling C sources
C compiler: gcc -fno-strict-aliasing -fno-common -dynamic -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes

creating /var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/var
creating /var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/var/folders
creating /var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/var/folders/yg
creating /var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq
creating /var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T
creating /var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8
creating /var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/src.macosx-10.4-x86_64-2.7
compile options: '-I. -I/var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/src.macosx-10.4-x86_64-2.7 -I/usr/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/include -I/usr/local/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c'
gcc: /var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/src.macosx-10.4-x86_64-2.7/fortranobject.c
In file included from /var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/src.macosx-10.4-x86_64-2.7/fortranobject.c:2:
In file included from /var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/src.macosx-10.4-x86_64-2.7/fortranobject.h:13:
In file included from /usr/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/include/numpy/arrayobject.h:4:
In file included from /usr/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:17:
In file included from /usr/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1804:
/usr/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: "Using deprecated NumPy API, disable it by "          "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-W#warnings]
#warning "Using deprecated NumPy API, disable it by " 
 ^
1 warning generated.
gcc: /var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/src.macosx-10.4-x86_64-2.7/Mainmodule.c
In file included from /var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/src.macosx-10.4-x86_64-2.7/Mainmodule.c:17:
In file included from /var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/src.macosx-10.4-x86_64-2.7/fortranobject.h:13:
In file included from /usr/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/include/numpy/arrayobject.h:4:
In file included from /usr/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:17:
In file included from /usr/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1804:
/usr/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: "Using deprecated NumPy API, disable it by "          "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-W#warnings]
#warning "Using deprecated NumPy API, disable it by " 
 ^
1 warning generated.
compiling Fortran 90 module sources
Fortran f77 compiler: /usr/local/bin/gfortran -Wall -g -ffixed-form -fno-second-underscore -fPIC -O3 -funroll-loops
Fortran f90 compiler: /usr/local/bin/gfortran -Wall -g -fno-second-underscore -fPIC -O3 -funroll-loops
Fortran fix compiler: /usr/local/bin/gfortran -Wall -g -ffixed-form -fno-second-underscore -Wall -g -fno-second-underscore -fPIC -O3 -funroll-loops
compile options: '-I. -I/var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/src.macosx-10.4-x86_64-2.7 -I/usr/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/include -I/usr/local/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c'
extra options: '-J/var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/ -I/var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/'
gfortran:f90: main.f90
compiling Fortran sources
Fortran f77 compiler: /usr/local/bin/gfortran -Wall -g -ffixed-form -fno-second-underscore -fPIC -O3 -funroll-loops
Fortran f90 compiler: /usr/local/bin/gfortran -Wall -g -fno-second-underscore -fPIC -O3 -funroll-loops
Fortran fix compiler: /usr/local/bin/gfortran -Wall -g -ffixed-form -fno-second-underscore -Wall -g -fno-second-underscore -fPIC -O3 -funroll-loops
compile options: '-I. -I/var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/src.macosx-10.4-x86_64-2.7 -I/usr/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/include -I/usr/local/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c'
extra options: '-J/var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/ -I/var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/'
gfortran:f90: /var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/src.macosx-10.4-x86_64-2.7/Main-f2pywrappers2.f90
/usr/local/bin/gfortran -Wall -g -Wall -g -undefined dynamic_lookup -bundle /var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/src.macosx-10.4-x86_64-2.7/Mainmodule.o /var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/src.macosx-10.4-x86_64-2.7/fortranobject.o /var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/main.o /var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/src.macosx-10.4-x86_64-2.7/Main-f2pywrappers2.o -L. -L/usr/local/lib/gcc/x86_64-apple-darwin/4.8.2 -llibtest -lgfortran -o ./Main.so
ld: library not found for -llibtest
collect2: error: ld returned 1 exit status
ld: library not found for -llibtest
collect2: error: ld returned 1 exit status
error: Command "/usr/local/bin/gfortran -Wall -g -Wall -g -undefined dynamic_lookup -bundle /var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/src.macosx-10.4-x86_64-2.7/Mainmodule.o /var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/src.macosx-10.4-x86_64-2.7/fortranobject.o /var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/main.o /var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/var/folders/yg/65v1lbd153v1j

Answer 1

Your command:

gfortran -c -fPIC libtest.f90

produces an object file with position independent code. This is a pre-requisite of a shared library, not a shared library.

If you want to use the object as is, you can modify your f2py invocation:

f2py -c --fcompiler=gfortran -I. libtest.o -m Main main.f90

This will link the object file and produce the file Main.cpython-33.so (the python version number may differ for you) and you can then import main in your python code.

---

If you would rather actually produce a shared object, you need to compile to a shared library. One way to do this is:

gfortran -shared -O2 -o libtest.so -fPIC libtest.f90

This produces libtest.so and now your original f2py command will work with one small change:

f2py -c --fcompiler=gfortran -L. -I. -ltest -m Main main.f90

The small change I am referring to is changing -llibtest to -ltest, as the -l option will add lib to the front of the library and .so to the end, e.g. -ltest will look for libtest.so. This produces Main.cpython-33.so with a dynamic link dependency to libtest.so, so you will need to distribute both shared libraries in order to use the python module.