libraries - MatLab error: cannot open with static TLS

Question

Since a couple of days, I constantly receive the same error while using MATLAB which happens at some point with dlopen. I am pretty new to MATLAB, and that is why I don't know what to do. Google doesn't seem to be helping me either. When I try to make an eigenvector, I get this:

Error using eig
LAPACK loading error:
dlopen: cannot load any more object with static TLS

I also get this while making a multiplication:

Error using  * 
BLAS loading error:
dlopen: cannot load any more object with static TLS

I did of course look for the solutions to this problem, but I don't understand too much and don't know what to do. These are threads I found:

1. How do I use the BLAS library provided by MATLAB?
2. http://www.mathworks.de/de/help/matlab/matlab_external/calling-lapack-and-blas-functions-from-mex-files.html

Can someone help me please?

Examples of function calls demonstrating this error

>> randn(3,3)

ans =

 2.7694    0.7254   -0.2050             
-1.3499   -0.0631   -0.1241             
 3.0349    0.7147    1.4897            

>> eig(ans)

Error using eig
LAPACK loading error:
dlopen: cannot load any more object with static TLS

Answer 1

That's bug no 961964 of MATLAB known since R2012b (8.0). MATLAB dynamically loads some libs with static TLS (thread local storage, e.g. see gcc compiler flag -ftls-model). Loading too many such libs => no space left.

Until now mathwork's only workaround is to load the important(!) libs first by using them early (they suggest to put "ones(10)*ones(10);" in startup.m). I better don't comment on this "solution strategy".

Since R2013b (8.2.0.701) with Linux x86_64 my experience is: Don't use "doc" (the graphical help system)! I think this doc-utility (libxul, etc.) is using a lot of static TLS memory.

Here is an update (2013/12/31)

All the following tests were done with Fedora 20 (with glibc-2.18-11.fc20) and Matlab 8.3.0.73043 (R2014a Prerelease).

For more information on TLS, see Ulrich Drepper, ELF handling For Thread-Local Storage, Version 0.21, 2013, currently available at Akkadia and Redhat.

What happens exactly?

MATLAB dynamically (with dlopen) loads several libraries that need tls initialization. All those libs need a slot in the dtv (dynamic thread vector). Because MATLAB loads several of these libs dynamically at runtime at compile/link time the linker (at mathworks) had no chance to count the slots needed (that's the important part). Now it's the task of the dynamic lib loader to handle such a case at runtime. But this is not easy. To cite dl-open.c:

For static TLS we have to allocate the memory here and now. This includes allocating memory in the DTV. But we cannot change any DTV other than our own. So, if we cannot guarantee that there is room in the DTV we don't even try it and fail the load.

There is a compile time constant (called DTV_SURPLUS, see glibc-source/sysdeps/generic/ldsodefs.h) in the glibc's dynamic lib loader for reserving a number of additional slots for such a mess (dynamically loading libs with static TLS in a multithreading program). In the glibc-Version of Fedora 20 this value is 14.

Here are the first libs (running MATLAB) that needed dtv slots in my case:

matlabroot/bin/glnxa64/libut.so
/lib64/libstdc++.so.6
/lib64/libpthread.so.0
matlabroot/bin/glnxa64/libunwind.so.8
/lib64/libuuid.so.1
matlabroot/sys/java/jre/glnxa64/jre/lib/amd64/server/libjvm.so
matlabroot/sys/java/jre/glnxa64/jre/lib/amd64/libfontmanager.so
matlabroot/sys/java/jre/glnxa64/jre/lib/amd64/libt2k.so
matlabroot/bin/glnxa64/mkl.so
matlabroot/sys/os/glnxa64/libiomp5.so
/lib64/libasound.so.2
matlabroot/sys/jxbrowser/glnxa64/xulrunner/xulrunner-linux-64/libxul.so
/lib64/libselinux.so.1
/lib64/libpixman-1.so.0
/lib64/libEGL.so.1
/lib64/libGL.so.1
/lib64/libglapi.so.0

Yes more than 14 => too many => no slot left in the dtv. That's what the error message tries to tell us and especially mathworks.

For the record: In order not to violate MATLAB's license I didn't debug, decompile or disassemble any part of the binaries shipped with MATLAB. I only debugged the free and open glibc-binaries of Fedora 20 that MATLAB were using to dynamically load the libs.

What can be done, to solve this problem?

There are 3 options:

(a) Rebuild MATLAB and do not dynamically load those libs (with initial-exec tls model) instead link against them (then the linker can count the required slots!)

(b) Rebuild those libs and ensure they are NOT using the initial-exec tls model.

(c) Rebuild glibc and increase DTV_SURPLUS in glibc/sysdeps/generic/ldsodefs.h

Obviously options (a) and (b) can only be done by mathworks.

For option (c) no source of MATLAB is needed and thus can be done without mathworks.

What is the status at mathworks?

I really tried to explain this to the "MathWorks Technical Support Department". But my impression is: they don't understand me. They closed my support ticket and suggested a telephone(!) conversation in January 2014 with a technical support manager.

I'll do my very best to explain this, but to be honest: I'm not very confident.

Update (2014/01/10): Currently mathworks is trying option (b).

Update (2014/03/19): For the file libiomp5.so you can download a newly compiled version (without static TLS) at mathworks, bug report 961964. And the other libs? No improvement there. So don't be suprised to get "dlopen: cannot load any more object with static TLS" with "doc", e.g. see bug report 1003952.