spring boot - SSLHandshakeException when trying to access ES instance from Docker

Question

I am trying to access 6.x ES instance using High Level REST Client 6.7.2. Access to this ES instance is provided to me via hostname (https://****.azureedge.net), username & password.

My Spring Boot application is getting data from the same ES without issues when it runs from my dev environment (IDE), but throws SSLHandshakeException as soon as I try run it from Docker container (from my development machine or K8s cluster in cloud).

Container is made with base image: FROM debian:stretch-slim & OpenJDK 11.0.2 with Spring Boot necessary modules.

I made some progress debugging with -Djavax.net.debug=all. It turns out that while running in docker image just few first steps of usual SSL handshaking happen:

Produced ClientHello handshake message
WRITE: TLS13 handshake, length = 2352
Raw write
Raw read (0000: 15 03 03 00 02 02 28    ......(   )
READ: TLSv1.2 alert, length = 2
Received alert message (
  "Alert": {
    "level"      : "fatal",
    "description": "handshake_failure"
  }
)

followed by SSLHandshakeException:

javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:938)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:227)
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1764)
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1749)
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1708)
at org.elasticsearch.client.SecurityClient.getSslCertificates(SecurityClient.java:508) 
....
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
at java.base/sun.security.ssl.Alert.createSSLException(Unknown Source)
at java.base/sun.security.ssl.Alert.createSSLException(Unknown Source)
at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source)
at java.base/sun.security.ssl.Alert$AlertConsumer.consume(Unknown Source)
at java.base/sun.security.ssl.TransportContext.dispatch(Unknown Source)
at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source)
at java.base/sun.security.ssl.SSLEngineImpl.decode(Unknown Source)
at java.base/sun.security.ssl.SSLEngineImpl.readRecord(Unknown Source)
at java.base/sun.security.ssl.SSLEngineImpl.unwrap(Unknown Source)
at java.base/sun.security.ssl.SSLEngineImpl.unwrap(Unknown Source)
at java.base/javax.net.ssl.SSLEngine.unwrap(Unknown Source)
at org.apache.http.nio.reactor.ssl.SSLIOSession.doUnwrap(SSLIOSession.java:271)
at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:316)
at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:509)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:120)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)

When running from my local environment handshake looks uninterrupted:

Produced ClientHello handshake message
WRITE: TLS13 handshake, length = 460
Raw write
Raw read
READ: TLSv1.2 handshake, length = 155
Consuming ServerHello
ServerHello
Negotiated protocol version: TLSv1.3
Session initialized:  Session(1560119025211|TLS_AES_256_GCM_SHA384)
WRITE: TLS13 change_cipher_spec, length = 1
Raw write
Raw read
READ: TLSv1.2 change_cipher_spec, length = 1
Consuming ChangeCipherSpec message
Raw read
READ: TLSv1.2 application_data, length = 27
...
Raw read
READ: TLSv1.2 application_data, length = 8469
Consuming server Certificate handshake message
... // here is the list of 3 certificates with "SHA256withRSA", "SHA256withRSA", "SHA1withRSA" signature algorithms
Found trusted certificate ? SHA1withRSA
...

While running locally I noticed CN=Microsoft IT TLS CA 2, OU=Microsoft IT, O=Microsoft Corporation, L=Redmond, ST=Washington, C=US, as well as CN=Baltimore CyberTrust Root, OU=CyberTrust, O=Baltimore, C=IE as issuers, maybe this is important, but I guess it is expected considering ES host address (Azure).

At the end I wanted to emphasise that I didn't need to do anything special to make this work in my macOS Java 11.0.2 development environment.

I already tried following, but that didn't change anything:

• Changing base Docker image from "slim" to non slim version
• Using OpenJDK 11.0.1 or 11.0.2
• Added cert from host to TrustStore that JVM is using in a runtime. (I checked in Docker container that there is indeed one more cert, but considering when handshake failure happens I imagine this is irrelevant)
• tried to enforce App with: "-Dcom.sun.net.ssl.enableECC=false", "-Djdk.tls.client.protocols=TLSv1.3", "-Dhttps.protocols=TLSv1.3", didn't help

Interesting: curl from Docker image with BasicAuth "talks" with same URL without issues (handshake completes) & small query returns results. I guess that curl and JVM are using different sources of trusted CAs inside the Docker, different algorithms for handshaking etc..

Thanks in advance for any help

Answer 1

TLDR: enforcing TLSv1.2 for client in the app made handshaking complete from inside the docker

After a lot of try/fail attempts I made it work. Following things didn't make any difference:

• using non "slim" debian base image insead of "slim"
• using OpenJDK 11.0.2 instead of 11.0.1
• adding host's certificate to JVM TrustedStore while building docker image so it is available when container starts.
• enforcing com.sun.net.ssl.enableECC=false
• enforcing TLSv1.3 for https.protocols and/or jdk.tls.client.protocols
• enforcing TLSv1.2 for https.protocols

What fixed handshake with host, was enforcing TLSv1.2 for client by using -Djdk.tls.client.protocols=TLSv1.2 in Dockerfile, so app runs with this flag inside container. This allowed SSL handshake to complete as it should work anyways. For some reason actual negotiation about protocol version didn't work without enforcing lower version of protocol for client. Logs from local vs docker environment doesn't show any difference but this helped in docker.

What helped me to find out was:

• setting of javax.net.debug=ssl:handshake or even more detailed javax.net.debug=all so I could see details of handshake attempts
• confirming that "at least someone" can establish outbound communication from inside the docker by using curl to send same request as app is trying, which worked because curl somehow figured out how to proceed handshaking with host.
• pure luck

Thanks everyone for support & ideas