selenium - How to parse data with looping over search field and appending output in a data set in python?

Question

My question is related to the previous question: How to parse several attributes of website with same class name in python?

I want to include the parsing within a loop over cap and append the resulting parsed text in a vector or data set at the end of the loop and then continue at the top.

My loop now looks like this:

driver = webdriver.Chrome('pathtoChrome/chromedriver.exe')
caps = ['11100']
for cap in caps:
   driver.get("https://www.conad.it/")
   WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[@href='javascript:void(0)']"))).click() # accept the cookies
   WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[@id='location-input']"))).send_keys(caps)
   WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//input[@class = 'btn btn-default btn-lg btn-block']"))).find_element_by_xpath("//input[@class = 'btn btn-default btn-lg btn-block']").click()
   #WebDriverWait(driver, 20).until(EC.element_to_be_clickable(driver.find_element_by_xpath("//input[@class = 'btn btn-default btn-lg btn-block']").click()
   print([item.text for item in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[contains(@class,'col-md-8')]//p")))])

The out-commented line is just another try. In both of my attempts, when I include the .click() command line within the loop, the CAP doesn't get answered.

It works however, if I do not loop, i.e.:

driver = webdriver.Chrome('pathtoChrome/chromedriver.exe')
driver.get("https://www.conad.it/")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[@href='javascript:void(0)']"))).click() # accept the cookies
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[@id='location-input']"))).send_keys('11100')
driver.find_element_by_xpath("//input[@class = 'btn btn-default btn-lg btn-block']").click()
print([item.text for item in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[contains(@class,'col-md-8')]//p")))])

I want to write the results in a data set or vector and then append the next round of the loop to it, something like this, which should append text to data found by typing 11100 or11020, but should not print anything when typing 11000, since there is no entry for this cap:

driver = webdriver.Chrome('pathtoChrome/chromedriver.exe')
caps = ['11000', '11100', '11020']
data = []

for cap in caps:
   driver.get("https://www.conad.it/")
   WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[@href='javascript:void(0)']"))).click() # accept the cookies
   WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[@id='location-input']"))).send_keys(caps)
   WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//input[@class = 'btn btn-default btn-lg btn-block']"))).find_element_by_xpath("//input[@class = 'btn btn-default btn-lg btn-block']").click()
   #WebDriverWait(driver, 20).until(EC.element_to_be_clickable(driver.find_element_by_xpath("//input[@class = 'btn btn-default btn-lg btn-block']").click() 
   print(data.append([item.text for item in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[contains(@class,'col-md-8')]//p")))]))

Any help is appreciated!

question from:https://stackoverflow.com/questions/65599537/how-to-parse-data-with-looping-over-search-field-and-appending-output-in-a-data

Answer 1

Use try..except block if item not found then continue the loop.

caps = ['11000','11100', '11020','13022']
data = []
driver.get("https://www.conad.it/")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[@href='javascript:void(0)']"))).click() # accept the cookies
for cap in caps:
   driver.get("https://www.conad.it/")
   WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[@id='location-input']"))).clear()
   WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[@id='location-input']"))).send_keys(cap)
   WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR,"input.btn.btn-default.btn-lg.btn-block"))).click()
   try:
      data.append([item.text for item in WebDriverWait(driver, 10).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[contains(@class,'col-md-8')]//p")))])

   except:
       print("no data found")
       continue

print(data)

Output:

[['Frazione Condemine 84, 11010 Sarre', 'Grand Chemin C/c Centreville 3, 11020 Saint-christophe', "Localita' Arensod 27, 11010 Sarre"], ['Grand Chemin C/c Centreville 3, 11020 Saint-christophe', "Localita' Perolle 21, 11024 Chatillon", 'Frazione Condemine 84, 11010 Sarre', "Localita' Arensod 27, 11010 Sarre"], ['Via Durio 26, 13019 Varallo', 'Via Brigate Garibaldi 24/a, 13019 Varallo']]