python - Chrome: API for performance data

Question

Problem Descritpion

How can I access the data from the Chrome Performance tool either from the browser console or using Chrome driver methods when using Selenium?

In particular, I am interested in getting the page loading time. On the top diagrams I can see that the last elements were loaded around 1700-1800 ms mark. The event log in the bottom shows that the last element started loading (and finished in a few ms) at 1720.1 ms from the time the page was reloaded.

What I Tried So Far

I tried using Performance Timing interface like this:

console.log(performance.timing.loadEventEnd - 
performance.timing.navigationStart)
357

But I guess it measures some other performance. loadEventEnd and navigationStart have the most extreme values among the attributes of timing and their difference is only 357 (ms).

I also tried logging performance info using Selenium like this (took some ideas from here and rewrote in Python):

import time
import json
import copy
from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
from selenium.webdriver.chrome.options import Options

caps = copy.deepcopy(DesiredCapabilities.CHROME)
caps['loggingPrefs'] = {'performance': 'ALL'}
chromedriver_options = Options()
chromedriver_options.add_argument("--headless")
chromedriver_options.add_argument("--window-size=1920,1080")
chromedriver_options.add_argument("--log-level=3") # suppress selenium logging to stdout
driver = webdriver.Chrome(chrome_options=chromedriver_options, executable_path='chromedriver.exe', desired_capabilities=caps)
driver.set_window_position(0, 0)
driver.implicitly_wait(8) # set timeout to 8 seconds
driver.get(my_url)
time.sleep(5) # wait for page to load for the first time
_ = driver.get_log('performance') # discard all previous logs
driver.refresh()
time.sleep(5) # make sure page loaded completely before writing log
with open('logs.txt', 'a') as f:
    for entry in driver.get_log('performance'):
        f.write(str(entry) + '
')
driver.close()

The resulting logs contain entries with calls to Page.frameStartedLoading and Page.frameStoppedLoading methods and their timestamps and I had an idea to measure the time between the first Page.frameStartedLoading and the last Page.frameStoppedLoading after flushing the log and calling driver.refresh().

Unfortunately, the resulting log file doesn't make a lot of sense. The performance logging doesn't stop (I thought it should since the web page is loaded at some point and nothing is happening performance-wise). If I call time.sleep(5) after refreshing, I would get some logs with 5 calls to each Page.frameStartedLoading and Page.frameStoppedLoading. If I call time.sleep(7) I would get 7 calls to each Page.frameStartedLoading and Page.frameStoppedLoading. And the difference between timestamp of the last Page.frameStoppedLoading and the first Page.frameStartedLoading would be ~7 seconds in this last case which is clearly not what I wanted.

Anyone has experience with capturing this page loading performance info?

Answer 1

If you want to access the Performance Data from the Chrome Developer Tool using ChromeDriver you can use the following code block :

• Selenium-Python Snippet

  from selenium import webdriver
  
  driver=webdriver.Chrome(executable_path=r'C:UtilityBrowserDriverschromedriver.exe')
  driver.get("http://www.google.com")
  performance_data = driver.execute_script("return window.performance.getEntries();")
  print (performance_data)
  

• The parameters extracted are as follows :

  connectEnd                 Time when server connection is finished.
  connectStart               Time just before server connection begins.
  domComplete                Time just before document readiness completes.
  domContentLoadedEventEnd   Time after DOMContentLoaded event completes.
  domContentLoadedEventStart Time just before DOMContentLoaded starts.
  domInteractive             Time just before readiness set to interactive.
  domLoading                 Time just before readiness set to loading.
  domainLookupEnd            Time after domain name lookup.
  domainLookupStart          Time just before domain name lookup.
  fetchStart                 Time when the resource starts being fetched.
  loadEventEnd               Time when the load event is complete.
  loadEventStart             Time just before the load event is fired.
  navigationStart            Time after the previous document begins unload.
  redirectCount              Number of redirects since the last non-redirect.
  redirectEnd                Time after last redirect response ends.
  redirectStart              Time of fetch that initiated a redirect.
  requestStart               Time just before a server request.
  responseEnd                Time after the end of a response or connection.
  responseStart              Time just before the start of a response.
  timing                     Reference to a performance timing object.
  navigation                 Reference to performance navigation object.
  performance                Reference to performance object for a window.
  type                       Type of the last non-redirect navigation event.
  unloadEventEnd             Time after the previous document is unloaded.
  unloadEventStart           Time just before the unload event is fired.
  

Console Output :

Page Title is : Google
[{'connectEnd': 2725.146514015987, 'connectStart': 1503.1734234800108, 'decodedBodySize': 204837, 'domComplete': 7603.4821342458235, 'domContentLoadedEventEnd': 4588.4400826362535, 'domContentLoadedEventStart': 4576.987229310746, 'domInteractive': 4559.208601432438, 'domainLookupEnd': 1503.1734234800108, 'domainLookupStart': 1503.1734234800108, 'duration': 7623.300733238722, 'encodedBodySize': 61075, 'entryType': 'navigation', 'fetchStart': 1503.1734234800108, 'initiatorType': 'navigation', 'loadEventEnd': 7623.300733238722, 'loadEventStart': 7603.516579526861, 'name': 'document', 'nextHopProtocol': 'h2', 'redirectCount': 0, 'redirectEnd': 1501.686197816412, 'redirectStart': 22.057947498907886, 'requestStart': 2725.729247123413, 'responseEnd': 3624.7713441197043, 'responseStart': 3349.3679493549, 'secureConnectionStart': 2049.7916668355165, 'startTime': 0, 'transferSize': 61926, 'type': 'navigate', 'unloadEventEnd': 0, 'unloadEventStart': 0, 'workerStart': 0}, {'connectEnd': 3388.3040845619494, 'connectStart': 3388.3040845619494, 'decodedBodySize': 1730, 'domainLookupEnd': 3388.3040845619494, 'domainLookupStart': 3388.3040845619494, 'duration': 237.3373068328902, 'encodedBodySize': 1730, 'entryType': 'resource', 'fetchStart': 3388.3040845619494, 'initiatorType': 'img', 'name': 'https://www.google.co.in/logos/doodles/2018/doodle-snow-games-day-15-5907794870927360.4-s.png', 'nextHopProtocol': 'h2', 'redirectEnd': 0, 'redirectStart': 0, 'requestStart': 3439.3892730100556, 'responseEnd': 3625.6413913948395, 'responseStart': 3625.271003314041, 'secureConnectionStart': 0, 'startTime': 3388.3040845619494, 'transferSize': 2242, 'workerStart': 0}, {'connectEnd': 0, 'connectStart': 0, 'decodedBodySize': 0, 'domainLookupEnd': 0, 'domainLookupStart': 0, 'duration': 230.20794413542762, 'encodedBodySize': 0, 'entryType': 'resource', 'fetchStart': 3425.689371645131, 'initiatorType': 'css', 'name': 'https://ssl.gstatic.com/gb/images/i1_1967ca6a.png', 'nextHopProtocol': 'h2', 'redirectEnd': 0, 'redirectStart': 0, 'requestStart': 0, 'responseEnd': 3655.8973157805594, 'responseStart': 0, 'secureConnectionStart': 0, 'startTime': 3425.689371645131, 'transferSize': 0, 'workerStart': 0}, {'connectEnd': 0, 'connectStart': 0, 'decodedBodySize': 0, 'domainLookupEnd': 0, 'domainLookupStart': 0, 'duration': 969.9734406621285, 'encodedBodySize': 0, 'entryType': 'resource', 'fetchStart': 3502.3913129811713, 'initiatorType': 'script', 'name': 'https://www.gstatic.com/external_hosted/createjs/createjs-2015.11.26.min.js', 'nextHopProtocol': 'h2', 'redirectEnd': 0, 'redirectStart': 0, 'requestStart': 0, 'responseEnd': 4472.364753643298, 'responseStart': 0, 'secureConnectionStart': 0, 'startTime': 3502.3913129811713, 'transferSize': 0, 'workerStart': 0}, {'connectEnd': 3520.930168473896, 'connectStart': 3520.930168473896, 'decodedBodySize': 12232, 'domainLookupEnd': 3520.930168473896, 'domainLookupStart': 3520.930168473896, 'duration': 134.0411771046674, 'encodedBodySize': 12232, 'entryType': 'resource', 'fetchStart': 3520.930168473896, 'initiatorType': 'css', 'name': 'https://www.google.co.in/logos/2018/snowgames_bobsled/cta.png', 'nextHopProtocol': 'h2', 'redirectEnd': 0, 'redirectStart': 0, 'requestStart': 3525.3821197382813, 'responseEnd': 3654.971345578564, 'responseStart': 3625.955856548778, 'secureConnectionStart': 0, 'startTime': 3520.930168473896, 'transferSize': 12745, 'workerStart': 0}, {'connectEnd': 4527.024551785848, 'connectStart': 4527.024551785848, 'decodedBodySize': 842405, 'domainLookupEnd': 4527.024551785848, 'domainLookupStart': 4527.024551785848, 'duration': 2632.3253968704244, 'encodedBodySize': 265174, 'entryType': 'resource', 'fetchStart': 4527.024551785848, 'initiatorType': 'script', 'name': 'https://www.google.co.in/logos/2018/snowgames_bobsled/snowgames_bobsled18.2.js', 'nextHopProtocol': 'h2', 'redirectEnd': 0, 'redirectStart': 0, 'requestStart': 4528.843667863202, 'responseEnd': 7159.3499486562705, 'responseStart': 4946.163646693538, 'secureConnectionStart': 0, 'startTime': 4527.024551785848, 'transferSize': 265741, 'workerStart': 0}, {'connectEnd': 4558.896162471504, 'connectStart': 4558.896162471504, 'decodedBodySize': 0, 'domainLookupEnd': 4558.896162471504, 'domainLookupStart': 4558.896162471504, 'duration': 1005.3106518587082, 'encodedBodySize': 0, 'entryType': 'resource', 'fetchStart': 4558.896162471504, 'initiatorType': 'other', 'name': 'https://www.google.co.in/gen_204?s=webaft&atyp=csi&ei=N6mPWp6mC8H00ASJtoL4Ag&rt=wsrt.3384,aft.1175,prt.1175', 'nextHopProtocol': 'h2', 'redirectEnd': 0, 'redirectStart': 0, 'requestStart': 4560.575876470308, 'responseEnd': 5564.206814330209, 'responseStart': 5564.1444075857435, 'secureConnectionStart': 0, 'startTime': 4558.896162471504, 'transferSize': 358, 'workerStart': 0}, {'connectEnd': 4576.353841378264, 'connectStart': 4576.353841378264, 'decodedBodySize': 417606, 'domainLookupEnd': 4576.353841378264, 'domainLookupStart': 4576.353841378264, 'duration': 1903.6302084304214, 'encodedBodySize': 144417, 'entryType': 'resource', 'fetchStart': 4576.353841378264, 'initiatorType': 'script', 'name': 'https://www.google.co.in/xjs/_/js/k=xjs.s.en.smQ6-n1iGHA.O/m=sx,sb,cdos,cr,elog,hsm,jsa,r,d,csi/am=wCLkeMEAyP8JgogEKwgsQIpgGBA/rt=j/d=1/t=zcms/rs=ACT90oH8FZej9QmzW2qBqlOOQ7DASmnKAA', 'nextHopProtocol': 'h2', 'redirectEnd': 0, 'redirectStart': 0, 'requestStart': 4578.110550711152, 'responseEnd': 6479.984049808687, 'responseStart': 5564.372556917789, 'secureConnectionStart': 0, 'startTime': 4576.353841378264, 'transferSize': 144999, 'workerStart': 0}, {'connectEnd': 0, 'connectStart': 0, 'decodedBodySize': 0, 'domainLookupEnd': 0, 'domainLookupStart': 0, 'duration': 574.9679253647753, 'encodedBodySize': 0, 'entryType': 'resource', 'fetchStart': 4791.30009458269, 'initiatorType': 'script', 'name': 'https://www.gstatic.com/og/_/js/k=og.og2.en_US.SpCLDXmWlPM.O/rt=j/m=def/exm=in,fot/d=1/ed=1/rs=AA2YrTtMoJJMGQfOfYZyZ7reaiaiva99OQ', 'nextHopProtocol': 'h2', 'redirectEnd': 0, 'redirectStart': 0, 'requestStart': 0, 'responseEnd': 5366.2680199474635, 'responseStart': 0, 'secureConnectionStart': 0, 'startTime': 4791.30009458269, 'transferSize': 0, 'workerStart': 0}, {'connectEnd': 0, 'connectStart': 0, 'decodedBodySize': 0, 'domainLookupEnd': 0, 'domainLookupStart': 0, 'duration': 1586.0147296126488, 'encodedBodySize': 0, 'entryType': 'resource', 'fetchStart': 5417.5655534222005, 'initiatorType': 'script', 'name': 'https://apis.google.com/_/scs/abc-static/_/js/k=gapi.gapi.en.29tAKSAI8cc.O/m=gapi_iframes,googleapis_client,plusone/rt=j/sv=1/d=1/ed=1/am=IA/rs=AHpOoo82FxkTgGRAoVn-fgFU3zdQ5QIqEw/cb=gapi.loaded_0', 'nextHopProtocol': 'h2', 'redirectEnd': 0, 'redirectStart': 0, 'requestStart': 0, 'responseEnd': 7003.580283034848, 'responseStart': 0, 'secureConnectionStart': 0, 'startTime': 5417.5655534222005, 'transferSize': 0, 'workerStart': 0}, {'connectEnd': 6646.626267079796, 'connectStart': 6646.626267079796, 'decodedBodySize': 83168, 'domainLookupEnd': 6646.626267079796, 'domainLookupStart': 6646.626267079796, 'duration': 783.8056119062338, 'encodedBodySize': 27299, 'entryType': 'resource', 'fetchStart': 6646.626267079796, 'initiatorType': 'script', 'name': 'https://www.google.co.in/xjs/_/js/k=xjs.s.en.smQ6-n1iGHA.O/m=aa,abd,async,dvl,foot,fpe,ipv6,lu,m,mu,sf,sonic,d3l/am=wCLkeMEAyP8JgogEKwgsQIpgGBA/exm=sx,sb,cdos,cr,elog,hsm,jsa,r,d,csi/rt=j/d=1/ed=1/t=zcms/rs=ACT90oH8FZej9QmzW2qBqlOOQ7DASmnKAA?xjs=s1', 'nextHopProtocol': 'h2', 'redirectEnd': 0, 'redirectStart': 0, 'requestStart': 6648.111871788991, 'responseEnd': 7430.431878986027, 'responseStart': 7159.731278179281, 'secureConnectionStart': 0, 'startTime': 6646.626267079796, 'transferSize': 27880, 'workerStart': 0}, {'connectEnd': 0, 'connectStart': 0, 'decodedBodySize': 0, 'domainLookupEnd': 0, 'domainLookupStart': 0, 'duration': 930.0310980101976, 'encodedBodySize': 0, 'entryType': 'resource', 'fetchStart': 6668.677730761095, 'initiatorType': 'img', 'name': 'https://www.google.com/textinputassistant/tia.png', 'nextHopProtocol': 'h2', 'redirectEnd': 0, 'redirectStart': 0, 'requestStart': 0, 'responseEnd': 7598.708828771292, 'responseStart': 0, 'secureConnectionStart': 0, 'startTime': 6668.677730761095, 'transferSize': 0, 'workerStart': 0}, {'connectEnd': 7222.302954756732, 'connectStart': 7222.302954756732, 'decodedBodySize': 469, 'domainLookupEnd': 7222.302954756732, 'domainLookupStart': 7222.302954756732, 'duration': 208.68612730489625, 'encodedBodySize': 469, 'entryType': 'resource', 'fetchStart': 7222.302954756732, 'initiatorType': 'img', 'name': 'https://www.google.co.in/logos/2018/snowgames_bobsled/main-sprite.png', 'nextHopProtocol': 'h2', 'redirectEnd': 0, 'redirectStart': 0, 'requestStart': 7223.911346761861, 'responseEnd': 7430.989082061627, 'responseStart': 7430.7698479787905, 'secureConnectionStart': 0, 'startTime': 7222.302954756732, 'transferSize': 980, 'workerStart': 0}]

From this output you can easily pull out the performance stats of your interest.