Working in AWS Cloud9, my script is:
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import requests
options = Options()
options.binary_location = '/usr/bin/google-chrome'
driver = webdriver.Chrome(chrome_options=options, executable_path='/home/ec2-user/environment/downloads/chromedriver')
url = 'http://google.com'
driver.get(url)
soup = BeautifulSoup(driver.page_source, 'lxml')
# driver.quit()
print(soup)
However, when I run the file, the run configuration hangs until I keyboard interrupt with ctrl+c. Traceback returns this:
> Traceback (most recent call last): File
> "/home/ec2-user/environment/vpy36/lib/python3.6/dist-packages/hubspot-marketplace.py",
> line 9, in <module>
> driver = webdriver.Chrome(chrome_options=options, executable_path='/home/ec2-user/environment/downloads/chromedriver')
> File
> "/home/ec2-user/environment/vpy36/lib/python3.6/dist-packages/selenium/webdriver/chrome/webdriver.py",
> line 75, in __init__
> desired_capabilities=desired_capabilities) File "/home/ec2-user/environment/vpy36/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py",
> line 154, in __init__
> self.start_session(desired_capabilities, browser_profile) File "/home/ec2-user/environment/vpy36/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py",
> line 243, in start_session
> response = self.execute(Command.NEW_SESSION, parameters) File "/home/ec2-user/environment/vpy36/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py",
> line 310, in execute
> response = self.command_executor.execute(driver_command, params) File
> "/home/ec2-user/environment/vpy36/lib/python3.6/dist-packages/selenium/webdriver/remote/remote_connection.py",
> line 466, in execute
> return self._request(command_info[0], url, body=data) File "/home/ec2-user/environment/vpy36/lib/python3.6/dist-packages/selenium/webdriver/remote/remote_connection.py",
> line 490, in _request
> resp = self._conn.getresponse() File "/usr/lib64/python3.6/http/client.py", line 1331, in getresponse
> response.begin() File "/usr/lib64/python3.6/http/client.py", line 297, in begin
> version, status, reason = self._read_status() File "/usr/lib64/python3.6/http/client.py", line 258, in _read_status
> line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1") File "/usr/lib64/python3.6/socket.py", line 586, in readinto
> return self._sock.recv_into(b) KeyboardInterrupt
Just spent the last 4 hours trying to install chrome and beautifulsoup onto AWS Cloud 9. Very difficult.
Edit: I just let it run until it stopped by itself and returned Message: unknown error: Chrome failed to start: exited abnormally
Edit 2: Driver info: chromedriver=2.10.267518,
platform=Linux 4.9.85-38.58.amzn1.x86_64 x86_64
Google Chrome 66.0.3359.117
See Question&Answers more detail:
os