cookies - Python requests like a browser?

Question

I want to get a webdocument from 'https://www.fanfiction.net/s/5218118/1/', but I am sadly unable to replicate the behaviour of my browser - the server always sends me something along the lines of "please enable cookies" or "complete Captcha". Is there a way to send requests like a browser, so the server delivers me the same document as if I am a browser? I already googled and tried to integrate cookies and a fake Useragent. Here is my code:

import requests
from fake_useragent import UserAgent

url = 'https://www.fanfiction.net/s/5218118/1/'

ua = UserAgent()
S = requests.Session()

header = {'User-Agent':str(ua.chrome)}
res = S.get(url, headers=header)
cookies = dict(res.cookies)


response = S.get(url, headers=header, cookies=cookies)

Thanks already in advance! EDIT: I know that I could use selenium, but I do not want to always update my chromedriver, and also I do not want to waste performance on selenium.

question from:https://stackoverflow.com/questions/65626366/python-requests-like-a-browser

Answer 1

Saw your EDIT but just in case, ...

Simple example with selenium, that give you the storytext

from selenium import webdriver
from bs4 import BeautifulSoup


browser = webdriver.Chrome('C:Program FilesChromeDriverchromedriver.exe')
browser.get('https://www.fanfiction.net/s/5218118/1/')

soup=BeautifulSoup(browser.page_source, 'lxml')

print(soup.select_one('#storytext').get_text())

browser.close()

Edit

Edited based on your question and the fact that the site is protected by cloudflare to avoid ddos attacks.

You could extract the tag texts by selenium, but as in example above, I use beautifulsoup

You are right, inspect the html with developer tools and the tag part looks like this:

<span class="xgray xcontrast_txt">
  Rated: <a class="xcontrast_txt" href="https://www.fictionratings.com/" target="rating">Fiction T</a> - English - Romance/Adventure - Naruto U., Hinata H. - Chapters: 6 - Words: 14,894 - Reviews: <a href="/r/13747729/">5</a>
  - Favs: 29 - Follows: 24 - Updated:
  <span data-xutime="1610096566">33m ago</span>
  - Published: 
  <span data-xutime="1605552788">Nov 16, 2020</span>
  - id: 13747729 
  </span>