I'm trying to scrape this website using Python and Selenium, it requires you to select a date from drop-down box then click search to view the planning applications.
URL: https://services.wiltshire.gov.uk/PlanningGIS/LLPG/WeeklyList.
I have the code working to select the first index of the drop-down box and press search. How would I open multiple windows for all the date options in the drop-down box or go through them one by one so I can scrape it?
from selenium import webdriver
from selenium.webdriver.support.ui import Select
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument('--headless')
options.add_argument('--disable-gpu')
driver = webdriver.Chrome('/Users/weaabduljamac/Downloads/chromedriver',
chrome_options=options)
url = 'https://services.wiltshire.gov.uk/PlanningGIS/LLPG/WeeklyList'
driver.get(url)
select = Select(driver.find_element_by_xpath('//*[@id="selWeek"]'))
select.select_by_index(1)
button = driver.find_element_by_id('csbtnSearch')
button.click()
app_numbers = driver.find_element_by_xpath('//*[@id="form1"]/table/tbody/tr[1]/td[1]/a').text
print(app_numbers)
Drop-down box HTML:
<select class="formitem" id="selWeek" name="selWeek">
<option selected="selected" value="2018,31">Week commencing Monday 30 July 2018</option>
<option value="2018,30">Week commencing Monday 23 July 2018</option>
<option value="2018,29">Week commencing Monday 16 July 2018</option>
<option value="2018,28">Week commencing Monday 9 July 2018</option>
<option value="2018,27">Week commencing Monday 2 July 2018</option>
<option value="2018,26">Week commencing Monday 25 June 2018</option>
<option value="2018,25">Week commencing Monday 18 June 2018</option>
<option value="2018,24">Week commencing Monday 11 June 2018</option>
<option value="2018,23">Week commencing Monday 4 June 2018</option>
<option value="2018,22">Week commencing Monday 28 May 2018</option>
</select>
See Question&Answers more detail:
os 与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…