Timeout for python requests.get entire response

Question

I'm gathering statistics on a list of websites and I'm using requests for it for simplicity. Here is my code:

data=[]
websites=['http://google.com', 'http://bbc.co.uk']
for w in websites:
    r= requests.get(w, verify=False)
    data.append( (r.url, len(r.content), r.elapsed.total_seconds(), str([(l.status_code, l.url) for l in r.history]), str(r.headers.items()), str(r.cookies.items())) )

Now, I want requests.get to timeout after 10 seconds so the loop doesn't get stuck.

This question has been of interest before too but none of the answers are clean. I will be putting some bounty on this to get a nice answer.

I hear that maybe not using requests is a good idea but then how should I get the nice things requests offer. (the ones in the tuple)

Answer 1

Set the timeout parameter:

r = requests.get(w, verify=False, timeout=10) # 10 seconds

Changes in version 2.25.1

The code above will cause the call to requests.get() to timeout if the connection or delays between reads takes more than ten seconds. See: https://docs.python-requests.org/en/master/user/advanced/#timeouts