python - Can't generate and use a value of some token within headers on the fly

Question

I'm trying to create a script using requests module to fetch some content from a webpage. My intention is to get 200 status code when I send a post http requests with appropriate parameters. So, this is the site address and this is the form that I filled in to generate the results. I just chose two options from the two dropdowns within the from and hit the Select My vehicle button to populate the results.

The way I've tried below works only when I hardcode cookie and digital_token within the headers. Please note that the username and the password are static.

As the value of cookie and digital_token are dynamic, it is necessary to produce them on the fly, otherwise if I hardcode their values, they expire in few minutes.

How can I generate and use the value of digital_token on the fly?

I've tried with:

import requests
from bs4 import BeautifulSoup

base = 'https://www.avis.com.au/en/home'
URL = 'https://www.avis.com.au/webapi/reservation/vehicles'

payload = {"rqHeader":{"brand":"","locale":"en_AU"},"nonUSShop":False,"pickInfo":"SYD","pickCountry":"AU","pickDate":"26/01/2021","pickTime":"9:00 AM","dropInfo":"SYD","dropDate":"27/01/2021","dropTime":"9:00 AM","couponNumber":"","couponInstances":"","couponRateCode":"","discountNumber":"","rateType":"","residency":"AU","age":25,"wizardNumber":"","lastName":"","userSelectedCurrency":"","selDiscountNum":"","promotionalCoupon":"","preferredCarClass":"","membershipId":"","noMembershipAvailable":False,"corporateBookingType":"","enableStrikethrough":"true","picLocTruckIndicator":False}

headers = {
    'content-type': 'application/json',
    'cookie': 'JSESSIONID=7-PfFQTdLoe4Allx772MXjNRHg-proLZhT7Wkn5k.w04vprecmapp02; akaalb_production_config=~op=avis_au:avis-au-us-wdc|~rv=36~m=avis-au-us-wdc:0|~os=7f956ca2417c5e686d715889b6a30f65~id=227f46458e1d74f397bc3732493d29f5; SessionPersistence=PROFILEDATA%3A%3DauthorizableId%253Danonymous; datacenter=cwdc; visitorId=cwdc-A6dcb0c81-876f-4200-b8b4-cf01d6665ad9; _gcl_au=1.1.1537375693.1611743635; _ga=GA1.3.666679016.1611743635; _gid=GA1.3.1276090703.1611743635; IR_gbd=avis.com.au; _fbp=fb.2.1611743635775.709997675; __qca=P0-236558422-1611743636477; APISID=f4bf5c40-9387-4319-b347-67a23e62ef8a; DIGITAL_TOKEN=86c21933-e3fe-4867-855e-256991d6c099-02-cwdc-ho4905; IR_8968=1611781534175%7C0%7C1611781534175%7C%7C; _uetsid=27fa7d10608b11eb994047dca2e1810a; _uetvid=27faeca0608b11eb8fe3e90805023a70; _gat_UA-6997633-40=1',
    'digital_token': 'eyJjdHkiOiJKV1QiLCJlbmMiOiJBMjU2R0NNIiwiYWxnIjoiZGlyIn0..mPE_QHWlQV4uOqy6.3cB4yCaW_mvSRbigEXpvvB-Y_zxrixxddIdq0kzoNyQlJEGgsefnR-8cSHXxWgDWU6Tp5Wl6WFfOjfPETeh9gnFmQpokP2-vZ0pX6k288PAjqg7GbJoaiYo7fBsDAs3nQfu_lsRtvtoqh4ku5jptU8juVLYl7DuBzSs3KtmJobP6esWh8mrx0ezClpg1Qt0E4jpfmSYmCx0dbDgSyKYxp-fB3YD8DDBB1xrkCWFwCW4P4HQqtt6S1EYbW_FKsFmwI-wnyouSXfMXCCeEc8_Ib6PO1LBU3PaWfCGyzEW2M-Z8QeKLYXxp5GCmgQDRylm5JiVhIhWPDLh9TLcQGyWKABOR9wrM7FkUSUBh9fGX-O-HRc63AxDSZ5zn81Wo9roqPLt_BoAm-RYxbmSjxD298x8i10beEv8vu3IDjuqGmbldmEpvdST2gE_-KRKZAwKDcM8NCzU9LuG3Q3G6eSOJRL3jfMx37uqVpOSe9qMQEYBtMn_tUhKWmThsYLzvmH6bgvmCKaAwOgd2TzSI_kcieAidjOH-OJLEbJ9zrGNw6orGu6x7_UcC9r8fy5T8JbsNXrOFQE_LL3jkzNymlTXQRZOO51AOgoxuVrNJBM8TrzHftY7y5HVa_gQleLFG_uN_RGg3mr9fzKxjcGEHFt15RqqtY2V70LOEpQeKrPjC7B-D-UI0XHa3lOhBp3bIDz4tX-loFP3QOwn2e5Nmqk_O0pr9Lfzjbc0jnk-6iUIFsQzKl4G-eJwaUkN5n549BpeaFrpfwInNNloVdVTqDwGLYA-xcYnkNBozXFuNjkNwYjv66Zlwaj3_bC6sz-SySyu7rtwhdW85O8DREhj8I6xb-VvwDDGsb6-h0GgeCjGN7JZ4qkpZ4BvLyPu-YxeB6mz-xcO22-t9lyx8abABlm8oLdhOkYBDgA8x7exGzNMbUiDe_D61bsgnkTu6XIMKrm4HMOcT9ZsGcXkK2BZ909i_cXyTDpq26YLgTjEcH5Ob5NRC-eSAw4eeddDmeAxWlL4TixCESaKsbxC8sXRC7tURQDIasW6xLAAkVSTCnlkOV34luR3VOU6KaHvoa6RaAdKgFw5VXvCddFkhA_mqWhOO60Y1ow9wRWu65Z8M6VjqsU_rRvW8HMlfUqJ9I8EHQJ4yG3PcbNRxx4qzpXOs6TGsk9OMhQzPOaEoo4-hzaV4f54HeyZDFUx2gI8CEopbd1Shg1e0p1Mk0gEuM12550A--M6dvZ0oMLL4lv9XzEEe2mGy16gmiRsRkvEoFcumbM1XUuX3IkuCVHk9OcPNNlVPD2jIGT8prY8aHDP8aJ6wl0b1GtoiwDA6CAA9ePzRpLN4G617Wnotttmao5YVrcE_b1O0CQbfYqLQ_lPvTl1n4-P1p-YuiRwYAdw2BQEpOv0LyS1ekycEzWHN4-BOFr9MxQ.bZcUICoU7nJzaxeT0rENtg',
    'password': 'AVISCOM',
    'user-agent': 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.104 Safari/537.36',
    'username': 'AVISCOM'
}

with requests.Session() as s:
    s.headers.update(headers)
    r = s.post(URL,json=payload)
    print(r.status_code)

question from:https://stackoverflow.com/questions/65927752/cant-generate-and-use-a-value-of-some-token-within-headers-on-the-fly

Answer 1

As already mentioned within the comments, this token is JS generated and can not be fetched using requests or curl or whatever.

An easy (but not very performant) way is Selenium using e.g. Firefox:

from selenium import webdriver
from selenium.webdriver.firefox.options import Options

options = Options()
options.headless = True   # Don't open a visible pop-up window
driver = webdriver.Firefox(options=options)

driver.get('https://www.avis.com.au/en/home')
cookies = driver.get_cookies()

driver.quit()

print(cookies)

[{'name': 'akaalb_production_config', 'value': '~op=avis_au:avis-au-us-dal|~rv=37~m=avis-au-us-dal:0|~os=7f956ca2417c5e686d715889b6a30f65~id=a09a33378351c4e0fb1683926d7cf558', 'path': '/', 'domain': 'www.avis.com.au', 'secure': True, 'httpOnly': False}, {'name': 'APISID', 'value': 'd3879ba0-bdec-488e-94f5-0c517b8d300f', 'path': '/', 'domain': 'www.avis.com.au', 'secure': True, 'httpOnly': True}, {'name': 'DIGITAL_TOKEN', 'value': '89a7cd03-276c-4f2f-942f-07a5171a13d2-02-cdal-ho3904', 'path': '/', 'domain': 'www.avis.com.au', 'secure': True, 'httpOnly': True}, {'name': 'datacenter', 'value': 'cdal', 'path': '/', 'domain': 'www.avis.com.au', 'secure': True, 'httpOnly': True}, {'name': 'visitorId', 'value': 'cdal-A96653921-88e6-431a-9b08-2582dfe47a75', 'path': '/', 'domain': 'www.avis.com.au', 'secure': True, 'httpOnly': True, 'expiry': 1927796685}, {'name': 'SessionPersistence', 'value': 'PROFILEDATA%3A%3DauthorizableId%253Danonymous', 'path': '/', 'domain': 'www.avis.com.au', 'secure': False, 'httpOnly': False}]

Here you already see your token. Fetch it e.g. via:

print([ c for c in cookies if 'DIGITAL_TOKEN' in c.values() ][0]['value'])

89a7cd03-276c-4f2f-942f-07a5171a13d2-02-cdal-ho3904