Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Post an Article
Welcome To Ask or Share your Answers For Others

Categories

python 3.x - How to retrieve html source from the web application WindGuru

0 votes
369 views
in Technique[技术] by (71.8m points)

python 3.x - How to retrieve html source from the web application WindGuru

It's only happen when I use the headless mode.

from selenium import webdriver

options = webdriver.ChromeOptions()
options.add_argument('--headless')

driver = webdriver.Chrome(
    r'C:UsersUsuarioDownloadschromedriver.exe',
    chrome_options=options,
)
driver.get('https://www.windguru.cz/3640')
html = driver.page_source
driver.quit()

# Result
# html = <html xmlns="http://www.w3.org/1999/xhtml"><head></head><body></body></html>

What is happening?

See Question&Answers more detail:os

与恶龙缠斗过久,自身亦成为恶龙；凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Reply

0 votes
by (71.8m points)

You were almost there. You simply need to induce WebDriverWait for the desired element to be visible and you can use the following solution:

  • Code Block:

    from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

options = webdriver.ChromeOptions()
options.add_argument('--headless')
options.add_argument("start-maximized")
options.add_argument("disable-infobars")
options.add_argument("--disable-extensions")
driver = webdriver.Chrome(chrome_options=options, executable_path=r'C:UtilityBrowserDriverschromedriver.exe')
driver.get('https://www.windguru.cz/3640')
WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "#forecasts-page")))
print(driver.page_source)
driver.quit()

  • Console Output:

    <!DOCTYPE html><html xmlns="http://www.w3.org/1999/xhtml" class=" no-flash svg no-hiddenscroll no-touchevents hairline no-hires indexeddb" style=""><head>
<title>Windguru - Argentina - Mar del Plata</title>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="Robots" content="index,follow" />

<link rel="manifest" href="/manifest.json" />

<meta name="mobile-web-app-capable" content="yes" />
<meta name="apple-mobile-web-app-capable" content="yes" />
<meta name="application-name" content="Windguru" />
<meta name="apple-mobile-web-app-title" content="Windguru" />
<meta name="msapplication-navbutton-color" content="#333333" />
<meta name="apple-mobile-web-app-status-bar-style" content="black" />
<meta name="apple-mobile-web-app-title" content="Windguru" />
<meta name="msapplication-starturl" content="/" />

<meta name="theme-color" content="#333333" />

<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no" />

<link rel="icon" type="image/png" sizes="192x192" href="/img/windguru-icon-192x192.png" />
<link rel="apple-touch-icon" type="image/png" sizes="192x192" href="/img/windguru-icon-192x192.png" />


<meta name="viewport" content="width=device-width, initial-scale=1, user-scalable=1" /><meta name="description" content="Windguru weather forecast for Argentina - Mar del Plata. Special wind and weather forecast for windsurfing, kitesurfing and other wind related sports." />
<meta name="keywords" content="wind forecast, live wind, windsurfing, kitesurfing, kiteboarding, surfing, snowkiting, sailing, paragliding, wave, meteorology, numeric model, wind speed, GFS, WRF, NWW3, ICON, NAM, HRW, WAM, COSMO, GWAM, EWAM, weather stations, wind stations, forecast maps, Argentina - Mar del Plata" />
<meta name="google-signin-client_id" content="902500120979-b876d301np6vmv1hlcu9kn1rvgg8097o.apps.googleusercontent.com" />
<link rel="apple-touch-icon-precomposed" href="https://www.windguru.net/img/apple-touch-icon-precomposed.png" />
<link rel="icon" href="https://www.windguru.net/img/windguru-icon-196x196.png" />
<link href="https://fonts.googleapis.com/css?family=Open+Sans:300,400,600&amp;subset=latin,latin-ext" rel="stylesheet" type="text/css" />


<script async="" src="//www.google-analytics.com/analytics.js"></script><script src="https://www.gstatic.com/firebasejs/5.0.4/firebase-app.js"></script>
<script src="https://www.gstatic.com/firebasejs/5.0.4/firebase-messaging.js"></script>


<script>
  // Initialize Firebase
  var config = {
    apiKey: "AIzaSyDwc6isuJ-h__wL8NbBgRvIYuJ_IoG1ubM",
    authDomain: "windguru-1091.firebaseapp.com",
    databaseURL: "https://windguru-1091.firebaseio.com",
    projectId: "windguru-1091",
    storageBucket: "windguru-1091.appspot.com",
    messagingSenderId: "902500120979"
  };
  firebase.initializeApp(config);
</script>

<script>
  (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
  (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
  m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
  })(window,document,'script','//www.google-analytics.com/analytics.js','ga');

  ga('create', 'UA-885239-4', 'auto');
  ga('set', 'anonymizeIp', true);

ga(function(tracker) {
  window['WGGA_cid'] = tracker.get('clientId');
  window['WGGA_tid'] = tracker.get('trackingId');
});  
  //ga('send', 'pageview');

</script>
<script type="text/javascript">
var WgLang = {"legend":{"SMER":"Wind direction","TMP":"Temperature","WINDSPD":"Wind speed","MWINDSPD":"Modif. wind","APCP":"Rain (mm/3h)","TCDC":"Cloud cover (%)","HTSGW":"Wave","WAVESMER":"Wave direction","RATING":"Windguru rating","PERPW":"Wave period (s)","APCP1":"Rain (mm/1h)","GUST":"Wind gusts","SLP":"&lt;span class="helpinfhpa"&gt;*Pressure (hPa)&lt;/span&gt;","RH":"Humidity (%)","FLHGT":"&lt;span class="helpinffl"&gt;*0u00b0 isotherm (m)&lt;/span&gt;","CDC":"Cloud cover (%)&lt;br/&gt;high / mid / low","TMPE":"&lt;span class="helpinftmp"&gt;*Temperature &lt;/span&gt;","WCHILL":"Wind chill","APCPs":"&lt;span class="helpinfsnow"&gt;*Precip. (mm/3h)&lt;/span&gt;","APCP1s":"&lt;span class="helpinfsnow"&gt;*Precip. (mm/1h)&lt;/span&gt;","WVHGT":"Wind wave","WVPER":"Wind wave per.(s)","WVDIR":"Wind wave dir.","SWELL1":"Swell","SWPER1":"Swell period (s)","SWDIR1":"Swell direction","SWELL2":"2.Swell","SWPER2":"2.Swell period (s)","SWDIR2":"2.Swell dir.","DIRPW":"Wave direction","WAVEDIR":"Wave direction"},"tooltip":{"TMPE":"Temperature at 2 meters above surface adjusted to real altitude of the spot. More info in Help/FAQ section.","SLP":"Sea level pressure in hPa, values above 1000 hPa are printed &lt;b&gt;as x-1000&lt;/b&gt;","FLHGT":"Freezing level height in meters","sst":"Sea surface temperature based on satellite data. Valid for oceans and large lakes, more info in help/FAQ","APCP1s":"Precipitation in milimeters. Bold blue numbers indicate snowfall.","APCPs":"Precipitation in milimeters. Bold blue numbers indicate snowfall."},"dir":["N","NNE","NE","ENE","E","ESE","SE","SSE","S","SSW","SW","WSW","W","WNW","NW","NNW"],"weekday":["Su","Mo","Tu","We","Th","Fr","Sa"],"txt":{"archive":"Archive","tides":"Tides","detail":"Detail / Map","link":"Link","timezone":"Timezone","help":"Help","options":"Options","choose_m":"Choose wind modification","loading":"Loading forecast...","delayed":"12 hours delayed forecast. Latest WRF forecasts are only available to Windguru PRO subscribers. &lt;a href='help_index.php?sec=pro'&gt;Click for more info.&lt;/a&gt;","delayed_short":"12 hours delayed forecast. Latest WRF forecasts are only available to Windguru PRO subscribers.","custom_onlypro":"This forecast is only available to Windguru PRO subscribers","lastupdated":"Last updated","nextexpected":"Next update expected","timeleft":"Time left"},"tab":{"forecast":"Forecast","graph":"&lt;img src="/images/gricon.png" width="15" height="10"/&gt;","2d":"2D","2d_t":"Temperature (0 ... 5000 m)","2d_w":"Wind (0 ... 5000 m)","2d_t_l":"Temperature (alt ... +2000 m)","2d_w_l":"Wind (alt ... +2000 m)","map":"Map","webcams":"Webcams","reports":"Wind reports","accommodation":"Accommodation","schools":"Schools/Rentals","shops":"Shops","other":"Other...","directory":"Links","fcst_graph":"&lt;img src="/img/gricon.png"/&gt;","more":"&lt;span class="butt-txt"&gt;More&lt;/span&gt;","statistic":"Statistics","archive":"Archive","alert":"Add alert","share_widget":"Add to your website"},"units":{"kmh":"km/h","mph":"mph","ms":"m/s","msd":"m/s","knots":"knots","bft":"Bft","c":"&amp;deg;C","f":"&amp;deg;F","m":"m","ft":"ft"},"maps":{"windspd":"Wind","t2m":"Temperature","press":"Pressure","tcdc_apcp3":"Rain / clouds","tcdc_apcp1":"Rain / clouds"},"mapsi":{"windspd":"wind","t2m":"temperature","press":"pressure","tcdc_apcp3":"precipitation","tcdc_apcp1":"precipitation"},"gmap":{"link_f":"Forecast","link_a":"Archive","link_d":"Detail","link_add":"Add to favourites","link_s":"Select"},"spotmenu":{"sel_zeme":"SELECT COUNTRY","sel_spot":"SELECT SPOT","num_spot":"spots","num_reg":"regions","num_zeme":"countries","sel_all":"ALL","qs_hint":"Type spot name (min. 3 characters)"},"langdir":{"dir":"int"}};

</script>
<script type="text/javascript" src="https://beta.windguru.net/js/pak/bs.min.000024.js?WGNOEXP"></script>
<script type="text/javascript" src="https://beta.windguru.net/js/pak/wg.min.000112.js?WGNOEXP"></script>
<script type="text/javascript" src="https://beta.windguru.net/js/min/WG.Ads.min.000013.js?WGNOEXP"></script>
<link rel="stylesheet" href="https://beta.windguru.net/css/min/index.min.000048.css?WGNOEXP" type="text/css" />
<script type="text/javascript">
WG.cdn = 'https://www.windguru.net';
WG.forecast_api = 'https://www.windguru.net/int/iapi.php';
WG.consent_new = 1;
WG.user = new WG.User();

WG.camblacklist = [1427387412]
WG._user_properties = {"id_user":0,"email":null,"username":"","wj":"knots","tj":"c","waj":"m","odh":3,"doh":22,"wrap":40,"fhours":240,"limit1":10.63,"limit2":15.57,"limit3":19.41,"tlimit":10,"vt":"forecast","directory":0,"wrapnew":null,"cellsize":null,"default_set":0,"default_id_spot":53,"consent_timestamp":0,"allow_personal_ads":false,"pro":false,"news_id":18,"news_pulse":1};
WG.ads.on = true;
WG.ads.ajax = true;

</script>
<script async="" src="//pagead2.googlesyndication.com/pagead/js/adsbygoogle.js"></script>
<script src="https://www.windguru.net/js/src/mapbox-gl.js"></script>
<script src="https://www.windguru.net//js/src/leaflet-mapbox-gl.js"></script>
<link rel="stylesheet" href="https://www.windguru.net/css/src/mapbox-gl.css" />

<script type="text/javascript">
if(navigator.serviceWorker) {
  navigator.serviceWorker.register('/sw.php').catch(function(err) {
    console.error('Unable to register service worker.', err);
  });
}

</script>
<script type="text/javascript" src="https://beta.windguru.net/js/min/main.min.000070.js?WGNOEXP"></script>
<script>
$(function(){
     if(WG.isRunningiOSStandalone()) {
     WG.gae('standalone','ios');
     }
     if(WG.isRunningAndroidStandalone()) {
     WG.gae('standalone','android');
     }
});    
</script>
<link rel="preload" href="https://adservice.google.co.in/adsid/integrator.js?domain=www.windguru.cz" as="script" /><script type="text/javascript" src="https://adservice.google.co.in/adsid/integrator.js?domain=www.windguru.cz"></script><link rel="preload" href="https://adservice.google.com/adsid/integrator.js?domain=www.windguru.cz" as="script" /><script type="text/javascript" src="https://adservice.google.com/adsid/integrator.js?domain=www.windguru.cz"></script><style>@-webkit-keyframes jBox-animation-zoomOutOpen-open {0% {-webkit-transform:scale(1.1);}100% {-webkit-transform:scale(1);}}.jBox-animation-zoomOutOpen-open {-webkit-animation-duration: 180ms;-webkit-animation-name: jBox-animation-zoomOutOpen-open;}@-webkit-keyframes jBox-animation-zoomOutClose-close {0% {-webkit-transform:scale(1);}100% {-webkit-transform:scale(1.1);}}.jBox-animation-zoomOutClose-close {-webkit-animation-duration: 180ms;-webkit-animation-name: jBox-animation-zoomOutClose-close;}</style><style>@-webkit-keyframes jBox-animation-zoomOutOpen-open {0% {-webkit-transform:scale(1.1);}100% {-webkit-transform:scale(1);}}.jBox-animation-zoomOutOpen-open {-webkit-animation-duration: 180ms;-webkit-animation-name: jBox-animation-zoomOutOpen-open;}@-webk

与恶龙缠斗过久,自身亦成为恶龙；凝视深渊过久,深渊将回以凝视…
OGeek|极客中国-欢迎来到极客的世界，一个免费开放的程序员编程交流平台！开放，进步，分享！让技术改变生活，让极客改变未来！ Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

Just Browsing Browsing

1.4m articles

1.4m replys

5 comments

57.0k users

Most popular tags

Snow Theme by Q2A Market
Powered by Question2Answer
...