ssl - Is it possible to identify TLS info. in requests response?

Question

I am using python's requests module. I can get the server's response headers and application layer data as:

import requests
r = requests.get('https://yahoo.com')
print(r.url)  

My question: Does requests allow retrieving Transport layer data (the server's TLS selected version, ciphersuite, etc. ?).

Answer 1

Here is a quick ugly monkey patching version that works:

import requests
from requests.packages.urllib3.connection import VerifiedHTTPSConnection

SOCK = None

_orig_connect = requests.packages.urllib3.connection.VerifiedHTTPSConnection.connect

def _connect(self):
    global SOCK
    _orig_connect(self)
    SOCK = self.sock

requests.packages.urllib3.connection.VerifiedHTTPSConnection.connect = _connect

requests.get('https://yahoo.com')
tlscon = SOCK.connection
print 'Cipher is %s/%s' % (tlscon.get_cipher_name(), tlscon.get_cipher_version())
print 'Remote certificates: %s' % (tlscon.get_peer_certificate())
print 'Protocol version: %s' % tlscon.get_protocol_version_name()

This yields:

Cipher is ECDHE-RSA-AES128-GCM-SHA256/TLSv1.2
Remote certificates: <OpenSSL.crypto.X509 object at 0x10c60e310>
Protocol version: TLSv1.2

However it is bad because monkey patching and relying on a unique global variable, which also means you can not inspect what happens at redirect steps, and so on.

Maybe with some work that can be turned out as a Transport Adapter, to get the underlying connection as a property of the request (probably of the session or something). That may create leaks though, because in the current implementation the underlying socket is thrown away as quickly as possible (see How to get the underlying socket when using Python requests).

Update, now using a Transport Adapter

This works, and is in line with the framework (no global variable, should handle redirects, etc. there may something to do for proxies though, like adding an override for proxy_manager_for too), but it is a lot more code.

import requests
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.connectionpool import HTTPSConnectionPool
from requests.packages.urllib3.poolmanager import PoolManager


class InspectedHTTPSConnectionPool(HTTPSConnectionPool):
    @property
    def inspector(self):
        return self._inspector

    @inspector.setter
    def inspector(self, inspector):
        self._inspector = inspector

    def _validate_conn(self, conn):
        r = super(InspectedHTTPSConnectionPool, self)._validate_conn(conn)
        if self.inspector:
            self.inspector(self.host, self.port, conn)

        return r


class InspectedPoolManager(PoolManager):
    @property
    def inspector(self):
        return self._inspector

    @inspector.setter
    def inspector(self, inspector):
        self._inspector = inspector

    def _new_pool(self, scheme, host, port):
        if scheme != 'https':
            return super(InspectedPoolManager, self)._new_pool(scheme, host, port)

        kwargs = self.connection_pool_kw
        if scheme == 'http':
            kwargs = self.connection_pool_kw.copy()
            for kw in SSL_KEYWORDS:
                kwargs.pop(kw, None)

        pool = InspectedHTTPSConnectionPool(host, port, **kwargs)
        pool.inspector = self.inspector
        return pool


class TLSInspectorAdapter(HTTPAdapter):
    def __init__(self, inspector):
        self._inspector = inspector
        super(TLSInspectorAdapter, self).__init__()

    def init_poolmanager(self, connections, maxsize, block=False, **pool_kwargs):
        self.poolmanager = InspectedPoolManager(num_pools=connections, maxsize=maxsize, block=block, strict=True, **pool_kwargs)
        self.poolmanager.inspector = self._inspector


def connection_inspector(host, port, connection):
    print 'host is %s' % host
    print 'port is %s' % port
    print 'connection is %s' % connection
    sock = connection.sock
    sock_connection = sock.connection
    print 'socket is %s' % sock
    print 'Protocol version: %s' % sock_connection.get_protocol_version_name()
    print 'Cipher is %s/%s' % (sock_connection.get_cipher_name(), sock_connection.get_cipher_version())
    print 'Remote certificate: %s' % sock.getpeercert()



url = 'https://yahoo.com'
s = requests.Session()
s.mount(url, TLSInspectorAdapter(connection_inspector))
r = s.get(url)

Yes, there is a lot of confusion in naming between socket and connection: requests uses a "connection pool" that has a set of connections, which are in fact, for HTTPS, a PyOpenSSL WrappedSocket, which has itself an underlying real TLS connection (that is a PyOpenSSL Connection object). Hence the strange forms in connection_inspector.

But this returns the expected:

host is yahoo.com
port is 443
connection is <requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x10bb372d0>
socket is <requests.packages.urllib3.contrib.pyopenssl.WrappedSocket object at 0x10bb37410>
Protocol version: TLSv1.2
Cipher is ECDHE-RSA-AES128-GCM-SHA256/TLSv1.2
Remote certificate: {'subjectAltName': [('DNS', '*.www.yahoo.com'), ('DNS', 'add.my.yahoo.com'), ('DNS', '*.amp.yimg.com'), ('DNS', 'au.yahoo.com'), ('DNS', 'be.yahoo.com'), ('DNS', 'br.yahoo.com'), ('DNS', 'ca.my.yahoo.com'), ('DNS', 'ca.rogers.yahoo.com'), ('DNS', 'ca.yahoo.com'), ('DNS', 'ddl.fp.yahoo.com'), ('DNS', 'de.yahoo.com'), ('DNS', 'en-maktoob.yahoo.com'), ('DNS', 'espanol.yahoo.com'), ('DNS', 'es.yahoo.com'), ('DNS', 'fr-be.yahoo.com'), ('DNS', 'fr-ca.rogers.yahoo.com'), ('DNS', 'frontier.yahoo.com'), ('DNS', 'fr.yahoo.com'), ('DNS', 'gr.yahoo.com'), ('DNS', 'hk.yahoo.com'), ('DNS', 'hsrd.yahoo.com'), ('DNS', 'ideanetsetter.yahoo.com'), ('DNS', 'id.yahoo.com'), ('DNS', 'ie.yahoo.com'), ('DNS', 'in.yahoo.com'), ('DNS', 'it.yahoo.com'), ('DNS', 'maktoob.yahoo.com'), ('DNS', 'malaysia.yahoo.com'), ('DNS', 'mbp.yimg.com'), ('DNS', 'my.yahoo.com'), ('DNS', 'nz.yahoo.com'), ('DNS', 'ph.yahoo.com'), ('DNS', 'qc.yahoo.com'), ('DNS', 'ro.yahoo.com'), ('DNS', 'se.yahoo.com'), ('DNS', 'sg.yahoo.com'), ('DNS', 'tw.yahoo.com'), ('DNS', 'uk.yahoo.com'), ('DNS', 'us.yahoo.com'), ('DNS', 'verizon.yahoo.com'), ('DNS', 'vn.yahoo.com'), ('DNS', 'www.yahoo.com'), ('DNS', 'yahoo.com'), ('DNS', 'za.yahoo.com')], 'subject': ((('commonName', u'*.www.yahoo.com'),),)}

Other ideas:

1. You may remove a lot of code if you do monkey patching like in https://stackoverflow.com/a/22253656/6368697 with basically poolmanager.pool_classes_by_scheme['http'] = MyHTTPConnectionPool; but this is still monkey patching, and it is sad that PoolManager does not give a nice API for the pool_classes_by_scheme variable to be able to easily override it
2. PyOpenSSL ssl_context may be able to hold callbacks that will be called during the TLS handshake and get the underlying data; then in init_poolmanager you would just need to setup the ssl_context in kwargs before calling superclass; this example in https://gist.github.com/aiguofer/1eb881ccf199d4aaa2097d87f93ace6a <= or maybe not because in fact the structure comes from ssl.create_default_context and ssl is far less powerful than PyOpenSSL and I see no way to add callbacks using ssl, where they exist for PyOpenSSL. YMMV.

PS:

1. Once you find out you have _validate_conn that you can override as it gets the proper connection object, life is easier
2. And especially if you do the import on top correctly, you need to use the urllib3 packages that are distributed inside requests, not the "real" urllib3 otherwise you get a lot of strange errors, because the same methods in both do not have the same signatures...