ssl - SSL3_GET_SERVER_CERTIFICATE certificate verify failed on Python when requesting (only) *.google.com

Question

I have encountered a really strange bug that has to do with SSL and python to google.com (or more generally I think with domains that have multiple certificate chains). Whenever I try to do a request to https://*.google.com/whatever I get the following error:

SSLError: ("bad handshake: Error([('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')],)",) while doing GET request to URL: https://google.com/

What I have done so far

I have gone through many hoops trying to get this working and am resorting to posting to Stack Overflow now that I don't know what to do. Here is what I have tried:

1. Noticed that date returned a date that was 2 minutes behind the real time (potentially invalidating my cert). I fixed this assuming it would validate the cert. This did not fix the issue.

2. Found out that Python 2.7.9 backported some SSL libraries from Python 3. I upgraded from Python 2.7.6 to 2.7.9 assuming the updates (which include fixes listed in this thread: https://serverfault.com/questions/692110/error-with-python2-as-a-https-client-with-an-nginx-server-and-ssl-certificate-ch) would fix it. No luck, same error.

3. Obviously setting verify=False works, but we are not willing to budge on security, we need to get verify=True to work.

4. curl https://google.com also works as expected. This is how I know it has to do with Python.

Environment

$ python -V
Python 2.7.9

$ pip list | grep -e requests
requests (2.9.1)

$ uname-a  # ubuntu 14.04
Linux staging.example.com 3.13.0-48-generic #80-Ubuntu SMP Thu Mar 12 11:16:15 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Example

This is only happening for google domains over https. Here is an example:

$ ipython
Python 2.7.9 (default, Jan  6 2016, 21:37:32)
Type "copyright", "credits" or "license" for more information.

IPython 4.0.1 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import requests

In [2]: requests.get('https://facebook.com', verify=True)
Out[2]: <Response [200]>

In [3]: requests.get('https://stackoverflow.com', verify=True)
Out[3]: <Response [200]>

In [4]: requests.get('https://spotify.com', verify=True)
Out[4]: <Response [200]>

In [5]: requests.get('http://google.com', verify=True) # notice the http
Out[5]: <Response [200]>

In [6]: requests.get('https://google.com', verify=True)
---------------------------------------------------------------------------
SSLError                                  Traceback (most recent call last)
<ipython-input-6-a7fff1831944> in <module>()
----> 1 requests.get('https://google.com', verify=True)

/example/.virtualenv/example/lib/python2.7/site-packages/requests/api.pyc in get(url, params, **kwargs)
     65
     66     kwargs.setdefault('allow_redirects', True)
---> 67     return request('get', url, params=params, **kwargs)
     68
     69

/example/.virtualenv/example/lib/python2.7/site-packages/requests/api.pyc in request(method, url, **kwargs)
     51     # cases, and look like a memory leak in others.
     52     with sessions.Session() as session:
---> 53         return session.request(method=method, url=url, **kwargs)
     54
     55

/example/.virtualenv/example/lib/python2.7/site-packages/requests/sessions.pyc in request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
    466         }
    467         send_kwargs.update(settings)
--> 468         resp = self.send(prep, **send_kwargs)
    469
    470         return resp

/example/.virtualenv/example/lib/python2.7/site-packages/requests/sessions.pyc in send(self, request, **kwargs)
    574
    575         # Send the request
--> 576         r = adapter.send(request, **kwargs)
    577
    578         # Total elapsed time of the request (approximately)

/example/.virtualenv/example/lib/python2.7/site-packages/requests/adapters.pyc in send(self, request, stream, timeout, verify, cert, proxies)
    445         except (_SSLError, _HTTPError) as e:
    446             if isinstance(e, _SSLError):
--> 447                 raise SSLError(e, request=request)
    448             elif isinstance(e, ReadTimeoutError):
    449                 raise ReadTimeout(e, request=request)

SSLError: ("bad handshake: Error([('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')],)",)

Answer 1

I found a solution. There seems to be a major issue in the version of certifi that was running. I found this out from this (very long) GitHub issue: https://github.com/certifi/python-certifi/issues/26

TL;DR

pip uninstall -y certifi && pip install certifi==2015.04.28