redhat - Apache 2.4 start/stop throws "undefined symbol: ber_sockbuf_io_udp" error after configuring it with Shibboleth SP 3.2.0

Question

Operating System: Red Hat Enterprise Linux Server 7.9 (Maipo)
Apache version: Apache/2.4.46 (Unix)
Shibboleth version: 3.2.0

Error when trying to stop Apache (apachectl stop):

httpd: Syntax error on line 528 of <Apache>/conf/httpd.conf: Syntax error on line 13 of <Apache>/conf/myshib.conf: Cannot load /usr/lib64/shibboleth/mod_shib_24.so into server: /usr/lib64/libldap_r-2.4.so.2: undefined symbol: ber_sockbuf_io_udp

Line 528 of <Apache>/conf/httpd.conf:

Include conf/myshib.conf

Line 13 of <Apache>/conf/myshib.conf:

LoadModule mod_shib /usr/lib64/shibboleth/mod_shib_24.so

Attempted - https://superuser.com/questions/1283252/slappasswd-symbol-lookup-error-undefined-symbol-ber-sockbuf-io-udp

• But running "export LD_LIBRARY_PATH=/lib64:$LD_LIBRARY_PATH" did not resolve the issue.

Output of "ldd /usr/lib64/libldap_r-2.4.so.2" command:

    linux-vdso.so.1 =>  (0x00007ffd1c76e000)
    liblber-2.4.so.2 => /usr/lib64/liblber-2.4.so.2 (0x00007f5b3e87d000)
    libresolv.so.2 => /usr/lib64/libresolv.so.2 (0x00007f5b3e663000)
    libsasl2.so.3 => /usr/lib64/libsasl2.so.3 (0x00007f5b3e446000)
    libssl.so.10 => /usr/lib64/libssl.so.10 (0x00007f5b3e1d4000)
    libcrypto.so.10 => /usr/lib64/libcrypto.so.10 (0x00007f5b3dd71000)
    libssl3.so => /usr/lib64/libssl3.so (0x00007f5b3db14000)
    libsmime3.so => /usr/lib64/libsmime3.so (0x00007f5b3d8ec000)
    libnss3.so => /usr/lib64/libnss3.so (0x00007f5b3d5b8000)
    libnssutil3.so => /usr/lib64/libnssutil3.so (0x00007f5b3d388000)
    libplds4.so => /usr/lib64/libplds4.so (0x00007f5b3d184000)
    libplc4.so => /usr/lib64/libplc4.so (0x00007f5b3cf7f000)
    libnspr4.so => /usr/lib64/libnspr4.so (0x00007f5b3cd41000)
    libpthread.so.0 => /usr/lib64/libpthread.so.0 (0x00007f5b3cb25000)
    libdl.so.2 => /usr/lib64/libdl.so.2 (0x00007f5b3c921000)
    libc.so.6 => /usr/lib64/libc.so.6 (0x00007f5b3c553000)
    libcrypt.so.1 => /usr/lib64/libcrypt.so.1 (0x00007f5b3c31c000)
    libgssapi_krb5.so.2 => /usr/lib64/libgssapi_krb5.so.2 (0x00007f5b3c0cf000)
    libkrb5.so.3 => /usr/lib64/libkrb5.so.3 (0x00007f5b3bde6000)
    libk5crypto.so.3 => /usr/lib64/libk5crypto.so.3 (0x00007f5b3bbb3000)
    libcom_err.so.2 => /usr/lib64/libcom_err.so.2 (0x00007f5b3b9af000)
    libkrb5support.so.0 => /usr/lib64/libkrb5support.so.0 (0x00007f5b3b79f000)
    libz.so.1 => /usr/lib64/libz.so.1 (0x00007f5b3b589000)
    librt.so.1 => /usr/lib64/librt.so.1 (0x00007f5b3b381000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f5b3eceb000)
    libfreebl3.so => /usr/lib64/libfreebl3.so (0x00007f5b3b17e000)
    libkeyutils.so.1 => /usr/lib64/libkeyutils.so.1 (0x00007f5b3af7a000)
    libselinux.so.1 => /usr/lib64/libselinux.so.1 (0x00007f5b3ad53000)
    libpcre.so.1 => /usr/lib64/libpcre.so.1 (0x00007f5b3aaf1000)

Then attempted - https://unix.stackexchange.com/questions/193320/yum-corrupted-on-rhel-6

Output of "for lib in /lib64/.so.; do if nm -D $lib|grep ber_sockbuf_io_udp; then echo $lib; fi; done" command:

000000000020e020 D ber_sockbuf_io_udp
/lib64/liblber-2.4.so.2
000000000020e020 D ber_sockbuf_io_udp
/lib64/liblber-2.4.so.2.10.7
                 U ber_sockbuf_io_udp
/lib64/libldap-2.4.so.2
                 U ber_sockbuf_io_udp
/lib64/libldap-2.4.so.2.10.7
                 U ber_sockbuf_io_udp
/lib64/libldap_r-2.4.so.2
                 U ber_sockbuf_io_udp
/lib64/libldap_r-2.4.so.2.10.7

Output of "rpm -qf /lib64/liblber-2.4.so.2" command:

openldap-2.4.44-22.el7.x86_64

Tried to install 'openldap' but it already existed

yum install openldap-2.4.44-22.el7.x86_64
Loaded plugins: langpacks, product-id, search-disabled-repos
Package openldap-2.4.44-22.el7.x86_64 already installed and latest version
Nothing to do

Restarted entire server and re-installed 'openldap' using "yum reinstall openldap.x86_64" command but no luck.

Apache starts and returns a response when accessing https://hostname.domain.com but downloading the Shib Metadata XML file via Apache using URL https://hostname.domain.com/Shibboleth.sso/Metadata fails.

Apologies if I missed adding anything obvious.

Update:
Works when using older Shibboleth version 3.1.0

question from:https://stackoverflow.com/questions/65930690/apache-2-4-start-stop-throws-undefined-symbol-ber-sockbuf-io-udp-error-after

Answer 1

similar thread here: https://unix.stackexchange.com/questions/193320/yum-corrupted-on-rhel-6

In my case (RHEL7.8 + Apache 2.4 + Shibboleth 3.2) I was able to resolve the issue by replacing the /usr/lib64/libldap_r-2.4.so.2 library with the one from the Apache directory: <APACHE_ROOT>/HTTPServer/openldap/lib/libldap_r-2.4.so.2

Run: locate libldap_r-2.4.so.2 to find the location of the library. In my case I got:

/app/ptc/Windchill_12.0/HTTPServer/openldap/lib/libldap_r-2.4.so.2
/app/ptc/Windchill_12.0/HTTPServer/openldap/lib/libldap_r-2.4.so.2.10.12
/usr/lib/libldap_r-2.4.so.2
/usr/lib/libldap_r-2.4.so.2.10.7
/usr/lib64/libldap_r-2.4.so.2
/usr/lib64/libldap_r-2.4.so.2.10.7

I noticed that the lib used in the error was in /usr/lib64 dir. I replaced it and now ./apachectl -t reports "Syntax OK"

I don't know enough about Linux to explain what is going on here or what the correct fix is. This was my observation and resolved the issue, I believe my steps are a hack though.

Anyone with a more elegant, upgrade proof solution?

Reference: https://groups.google.com/g/repmgr/c/TS7QfYEoNoY

cd /usr/lib64/
ll | grep libldap
lrwxrwxrwx.  1 root root       21 Feb 11 16:42 libldap-2.4.so.2 -> libldap-2.4.so.2.10.7
-rwxr-xr-x.  1 root root   352512 Jun  6  2020 libldap-2.4.so.2.10.7
lrwxrwxrwx.  1 root root       23 Feb 11 16:42 libldap_r-2.4.so.2 -> libldap_r-2.4.so.2.10.7
-rwxr-xr-x.  1 root root   381328 Jun  6  2020 libldap_r-2.4.so.2.10.7

It does seem like /usr/lib64/libldap_r2.4.so.2 is just a symlink to libldap_r-2.4.so.2.10.7.

I wonder if there are missing or deprecated symbols in 2.10.7... Is there any way to tell the diff in the 2 versions?

UPDATE I noticed that you can make use of the LoadFile command in the Apache conf. Adding LoadFile <APACHE_ROOT>/HTTPServer/openldap/lib/libldap_r-2.4.so.2 in my 00-shib.conf file before the LoadModule mod_shib /usr/lib64/shibboleth/mod_shib_24.so entry resolved the issue.

This still seems like a workaround / hack to me and there may be an underlying issue with the libraries / different versions.