Unable to access site with python selenium webdriver

Question

I have been trying to build a universal scraper. But somehow there is some site that I am unable to access for some reason.

I have tried to use various options available on the internet to make sure that I avoided the bot detection flag but somehow the site apparently "detects" that I am a bot.

Here are the options I have been using.

```options.add_argument("--disable-blink-features")
options.add_argument("--disable-blink-features=AutomationControlled")
options.add_argument("user-data-dir=" + r'C:UsersJEGSTUDIOAppDataLocalGoogleChromeselenium')
options.add_argument("window-size=1280,800")
options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)```

I see and compare the cookies, and it looks like this site is using Cloudflare js based on the cookie naming.

https://support.cloudflare.com/hc/en-us/articles/200170136-Understanding-Cloudflare-Challenge-Passage-Captcha-

Here is the full code so you guys can try

```from selenium import webdriver

options = webdriver.ChromeOptions()

options.add_argument("--disable-blink-features")
options.add_argument("--disable-blink-features=AutomationControlled")
options.add_argument("user-data-dir=" + r'C:UsersJEGSTUDIOAppDataLocalGoogleChromeselenium')
options.add_argument("window-size=1280,800")
options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)

driver = webdriver.Chrome(options=options, executable_path=r'C:UsersJEGSTUDIOGochiScraping ProjectScraper - AITOPIA v2chromedriver88crack.exe')

driver.execute_script("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})")

driver.get("https://google.com")

input('Next site')

driver.get("https://www.harrods.com/")

input('enter to quit')

driver.quit()```

Any clue would be appreciated

question from:https://stackoverflow.com/questions/65870090/unable-to-access-site-with-python-selenium-webdriver

Answer 1

options.add_argument("--remote-debugging-port=9222")

options.add_argument(
    "user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36")
options.add_experimental_option("excludeSwitches", ["enable-automation"])



driver = webdriver.Chrome(options=options)

driver.maximize_window()


driver.get("https://www.harrods.com/")

Adding remote debugging port makes the site works