php - "tlsv1 alert internal error" during handshake

Question

I have a PHP script that checks URLs availability (basically, the script should return true for a given URL when the URL could be opened in browser and vice versa). There is an URL I stumbled upon: https://thepiratebay.gd/. This URL could be correctly opened in browser, but fsockopen() just fails with the SSL handshake errors. There are not many options for debugging fsockopen() in PHP, but while digging more into it, I found that I am also not able to connect to https://thepiratebay.gd/ using console openssl client:

openssl s_client -connect thepiratebay.gd:443 
CONNECTED(00000003)
39613:error:14077438:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert internal error:/SourceCache/OpenSSL098/OpenSSL098-50/src/ssl/s23_clnt.c:602:

This website seem to open fine using web browser or curl, however, I was not able to find a way to connect to it via openssl. Apparently, the server uses TLS 1.2 with ECDHE-ECDSA-AES128-GCM-SHA256 cipher, but even when I force those for openssl, it still fails:

openssl s_client -cipher ECDHE-ECDSA-AES128-GCM-SHA256 -connect thepiratebay.gd:443 -tls1_2
CONNECTED(00000003)
140735195829088:error:14094438:SSL routines:SSL3_READ_BYTES:tlsv1 alert internal error:s3_pkt.c:1256:SSL alert number 80
140735195829088:error:1409E0E5:SSL routines:SSL3_WRITE_BYTES:ssl handshake failure:s3_pkt.c:596:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 0 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : 0000
    Session-ID: 
    Session-ID-ctx: 
    Master-Key: 
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1432931347
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
---

I've tried various openssl versions: 0.9.8y, 1.0.1g, and the most recent 0.9.8zf and 1.0.2a. I've also tried to run this on at least 5 servers (CentOS, Debian, OSX) with no luck.

Every other website seem to be handled fine, here is an example of a successful handshake output:

openssl s_client -connect stackoverflow.com:443  -tls1
CONNECTED(00000003)
depth=1 /C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert SHA2 High Assurance Server CA
verify error:num=20:unable to get local issuer certificate
verify return:0
---
Certificate chain
 0 s:/C=US/ST=NY/L=New York/O=Stack Exchange, Inc./CN=*.stackexchange.com
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert SHA2 High Assurance Server CA
 1 s:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert SHA2 High Assurance Server CA
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV Root CA
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIGajCCBVKgAwIBAgIQCn1PE//Ffo4Be8tPBlsAZDANBgkqhkiG9w0BAQsFADBw
MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3
d3cuZGlnaWNlcnQuY29tMS8wLQYDVQQDEyZEaWdpQ2VydCBTSEEyIEhpZ2ggQXNz
dXJhbmNlIFNlcnZlciBDQTAeFw0xMzEwMjIxMjAwMDFaFw0xNjA3MDYxMjAwMDBa
MGoxCzAJBgNVBAYTAlVTMQswCQYDVQQIEwJOWTERMA8GA1UEBxMITmV3IFlvcmsx
<---skipped few lines--->
qOHCjaUIx7vKszN4cqbvyry/NdxYkPCC7S8Eks8NjSyppzRL79tU0Yr1MUhVEd6h
GjB2qDwvAGqyWmLz1Q/l82lZbXyBF26DVTJ3RFRUzzieyzKucaVgohI7HC2yyJ9Y
AsE7wvVK4odQI3fRjOsLRaXjFtpiaor0rERUxM4mg7jj05leRBkSazNjv2xvCL5/
Qqm5PN666tREQwvgvXZgg+ZlKWkFyOq6X3THstM6CC8DTGED0cb94WPQA4YTp9OQ
rS3+OedQN+Nlu80Sk8Y=
-----END CERTIFICATE-----
subject=/C=US/ST=NY/L=New York/O=Stack Exchange, Inc./CN=*.stackexchange.com
issuer=/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert SHA2 High Assurance Server CA
---
No client certificate CA names sent
---
SSL handshake has read 3956 bytes and written 426 bytes
---
New, TLSv1/SSLv3, Cipher is DHE-RSA-AES128-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : DHE-RSA-AES128-SHA
    Session-ID: 2E38670F2CABEF3D65FAC67DB6D2E00DBACA4519E50B463D57FCFF8410640BF5
    Session-ID-ctx: 
    Master-Key: 4C63E5502FF7DD36853048E775435A76CB1FDEB37104D6714B1C37D89482D8111B93574D2B3D7F38A1EEFF85D69F9F54
    Key-Arg   : None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000 - 39 a8 c2 5f c5 15 04 b3-20 34 af fe 20 8e 4d 6c   9.._.... 4.. .Ml
    0010 - 6e 63 f1 e3 45 fd 2a 2c-d9 3c 0d ac 11 ab c0 c9   nc..E.*,.<......
    0020 - ce 51 19 89 13 49 53 a0-af 87 89 b0 5d e2 c5 92   .Q...IS.....]...
    0030 - af e5 84 28 03 4e 1e 98-4c a7 03 d5 5f fc 15 69   ...(.N..L..._..i
    0040 - 7c 83 d2 98 7d 42 50 31-30 00 d7 a8 3c 85 88 a7   |...}BP10...<...
    0050 - cd c0 bb 45 c8 12 b1 c8-4b 76 3c 41 5e 47 04 b5   ...E....Kv<A^G..
    0060 - 60 67 22 76 60 bb 44 f3-4b 3d 3d 99 af 0e dd 0d   `g"v`.D.K==.....
    0070 - 13 95 db 94 90 c2 0f 47-26 04 65 6b 71 b2 f8 1c   .......G&.ekq...
    0080 - 31 95 82 8b 00 38 59 08-1e 84 80 dc da 04 5c f0   1....8Y........
    0090 - ae cc 2b ac 55 0f 39 59-0b 39 7d c7 16 b9 60 ef   ..+.U.9Y.9}...`.

    Start Time: 1432930782
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)

It is hard to believe that all these openssl versions have the same bug, so I'm thinking that I'm doing something wrong.

Can anyone advice how to connect to this particular website using openssl?

Answer 1

These two are a bad combination:

-cipher ECDHE-ECDSA-AES128-GCM-SHA256

And:

error:/SourceCache/OpenSSL098/OpenSSL098-50/src/ssl/s23_clnt.c

OpenSSL 0.9.8 does not have full EC support. And it does not support TLS 1.1 or 1.2. To get the AEAD cipher suites, you need to use TLS 1.2. That means you need OpenSSL 1.0.0 or above (IIRC).

OpenSSL 1.0.1 and 1.0.2 have them, so its probably better to use those versions.

---

 openssl s_client -connect thepiratebay.gd:443 ...

The command you are looking for is: openssl s_client -connect thepiratebay.gd:443 -tls1_2 -servername thepiratebay.gd -CAfile XXX. -servername enlists SNI.

When I hit the site, the server was certified by AddTrust External CA Root. When you hit the site, it was certified by DigiCert High Assurance EV Root CA. And when you hit the site again, it was certified by COMODO ECC Certification Authority.

The different CAs and configurations speak to a distributed site behind a load balancer, with each participating web server in a slightly different configuration.

---

In addition to multiple web servers and configurations, some of the web servers themselves are misconfigured. They are misconfigured because they do not send the chain required to build a path for validation.

The chain should include (1) the server certificate; (2) Subordinate CAs or intermediates that form the chain to the "root". For (2), there may be one or more intermediates.

The chain should not include the root. You have to have the root, and it must be trusted.

---

This website seem to open fine using web browser or curl, however, I was not able to find a way to connect to it via openssl...

This is because the browsers carry around a list of hundreds of Root CAs and Subordinate CAs due to web server misconfigurations :) The list includes AddTrust External CA Root, DigiCert High Assurance EV Root CA, and COMODO ECC Root Certificate Authority.

---

Can anyone advice how to connect to this particular website using openssl?

OK, for the OpenSSL command, you should use -CAfile. Usually, you just use something like openssl s_client -connect ... -CAfile DigiCertHighAssuranceEVRootCA.crt (for the server certified with DigiCert High Assurance EV Root CA). But that won't work in this case.

You have to create a single file with the required Root and Subordinate CAs. The file should be a concatenation of the Root CAs and Subordinate CAs in PEM format required to build a path to validate the server certificate. It looks like it will need at least 3 or 4 certificates.

Or, you could forgo building you own file, and use something like cacert.pem. But there is some risk in using the CA Zoo (my affectionate term for them). For some of the risks, see Is cacert.pem unique to my computer?.

Programmatically, you would use SSL_CTX_load_verify_locations in OpenSSL. The concatenated PEM file is passed though CAfile.

I'm not sure what you would use in PHP.

---

Related, cacert.pem has 155 roots and subordinates. Most of them are not needed to certify the site thepiratebay.gd:

$ cat cacert.pem | grep BEGIN | wc -l
     155

Hence the reason you want to restrict your CAfile to only those necessary to certify the site.

---

(comment) Not sure this is the correct thread to ask, but now I wonder if there is a way to skip some of these checks programmatically to reduce number of false negatives...

I would probably not forgo the checks. Now that you understand what's going on, it should be easier to work with the system rather than abandoning it.

To reiterate, either:

1. Use only necessary Root and Subordinate CAs

   1. You build it, concatenation of PEM certificates
   2. Create file piratebay-certs.pem
   3. Add necessary CAs

2. Use a CA Zoo with predifined trusted Root and Subordinate CAs

   1. You download it
   2. cacert.pem

The third option is to get the site to fix its web server configurations. But if it has not happened by now, it probably won't happen. (And it could be a design decision - the site may be using multiple CAs to ensure no one CA can DoS the site. But that does not address the incomplete chain).

---

And the more general observation:

I have a PHP script that checks URLs availability (basically, the script should return true for a given URL when the URL could be opened in browser and vice versa

Moving away from piratebay.gd in particular to checking random URLs, you will probably have to use cacert.pem. That's because a random sample of 1 million sites will likely use all of them.

If piratebay.gd still fails, then find out what is missing from cacert.pem, and then:

cat cacert.pem > my-expanded-cacert.pem
cat missing-cert.pem >> my-expanded-cacert.pem