python - Message: Error: Polling for changes failed: NetworkError when attempting to fetch resource while downloading file through Selenium and FirefoxProfile

Question

I am trying to download file from a url using selenium and Firefox on python3 but that give me an error in the geckodriver log file:

 (firefox:13723): Gtk-WARNING **: 11:12:39.178: Theme parsing error:       <data>:1:77: Expected ')' in color definition
 1546945960048  Marionette  INFO    Listening on port 40601
 1546945960132  Marionette  WARN    TLS certificate errors will be ignored for this session
     console.error: BroadcastService: 
      receivedBroadcastMessage: handler for
      remote-settings/monitor_changes
       threw error:
            Message: Error: Polling for changes failed: NetworkError when attempting to fetch resource..
            Stack:
                remoteSettingsFunction/remoteSettings.pollChanges@resource://services-settings/remote-settings.js:188:13

I use geckodriver verssion 0.22 and firefow version 65.0. Also am on UBUNTU 18 (only ssh) geckodriver is in the /usr/bin file and have all the needed right.

I have read on google that this might be because of the COPS. But I really get what the COPS are or how to do to fix them (if that is the real problem).

here my code:

from os import getcwd
from pyvirtualdisplay import Display
from selenium import webdriver

# start the virtual display
display = Display(visible=0, size=(800, 600))
display.start()

# configure firefox profile to automatically save csv files in the current directory
fp = webdriver.FirefoxProfile()
fp.set_preference("browser.download.folderList", 2)
fp.set_preference("browser.download.manager.showWhenStarting", False)
fp.set_preference("browser.download.dir", getcwd())
fp.set_preference("browser.helperApps.neverAsk.saveToDisk", "text/csv")

driver = webdriver.Firefox(firefox_profile=fp)
page = "https://www.thinkbroadband.com/download"
driver.get(page)
driver.find_element_by_xpath("//*[@id='main-col']/div/div/div[8]/p[2]/a[1]").click()

Do you guys have any idea ?

Answer 1

This error message...

Message: Error: Polling for changes failed: NetworkError when attempting to fetch resource..

...implies that there was a NetworkError while attempting to fetch resource.

Here the main issue probably is related to Cross-Origin Resource Sharing (CORS)

Cross-Origin Resource Sharing (CORS) is a mechanism that uses additional HTTP headers to tell a browser to let a web application running at one origin (domain) have permission to access selected resources from a server at a different origin. A web application makes a cross-origin HTTP request when it requests a resource that has a different origin (domain, protocol, and port) than its own origin.

An example of a cross-origin request: The frontend JavaScript code for a web application served from http://domain-a.com uses XMLHttpRequest to make a request for http://api.domain-b.com/data.json.

For security reasons, browsers restrict cross-origin HTTP requests initiated from within scripts. For example, XMLHttpRequest and the Fetch API follow the same-origin policy. This means that a web application using those APIs can only request HTTP resources from the same origin the application was loaded from, unless the response from the other origin includes the right CORS headers.

Modern browsers handle the client-side components of cross-origin sharing, including headers and policy enforcement. But this new standard means servers have to handle new request and response headers.

Solution

You need to induce WebDriverWait for the desired element to be clickable and you can use the following solution:

• Code Block:

  from selenium import webdriver
  from os import getcwd
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.support import expected_conditions as EC
  
  # configure firefox profile to automatically save csv files in the current directory
  fp = webdriver.FirefoxProfile()
  fp.set_preference("browser.download.folderList", 2)
  fp.set_preference("browser.download.manager.showWhenStarting", False)
  fp.set_preference("browser.download.dir", getcwd())
  fp.set_preference("browser.helperApps.neverAsk.saveToDisk", "text/csv")
  driver = webdriver.Firefox(firefox_profile=fp, executable_path=r'C:UtilityBrowserDriversgeckodriver.exe')
  driver.get("https://www.thinkbroadband.com/download")
  WebDriverWait(driver, 5).until(EC.element_to_be_clickable((By.XPATH, "//div[@class='specific-download-headline' and contains(., 'Extra Small File (5MB)')]//following::p[1]/a"))).click()
  

• Snapshot:

• Reference: How to resolve “TypeError: NetworkError when attempting to fetch resource.”