How can I make sure if some HTML elements are loaded for Selenium + Python?

Question

From this link, I assume the DOM should be loaded as a whole at first in RAM.

How DOM works/is loaded? (in HTML)

But then I test in Selenium with a timeout exception. It seems even the timeout exception is raised, some elements can already be found, so it is not an empty object.

But I am wondering, how can I make sure some elements are already loaded? E.g. the HTML example, how can I make sure all <th> elements are loaded? Given the fact that I actually do not know the number of the <th> elements.

Code trial :

driver = webdriver.Chrome()
driver.set_page_load_timeout(10)
try:
    driver.get(url)
    print('load success!')
except TimeoutException:
    print(self.page_source)

Sample HTML:

<table width="910" border="0" cellpadding="3" cellspacing="0" id="fth1_" class="fth1_" style="display: none; position: fixed; top: 29px; left: 99px;">
     <thead style="background-color: rgb(233, 233, 233);">
        <tr align="center">
           <th id="f13" style="width: 121px;"><a href="t/?i=614&amp;o=1">Symbol</a></th>
           <th id="f13" style="width: 267px;"><a href="t/?i=614&amp;o=2">Name</a></th>
        </tr>
     </thead>
</table>

Answer 1

As per your code trials you are using ChromeDriver and Chrome Browser to Automate the steps. As you have configured set_page_load_timeout(10) the timeout exception is raised as the page havn't completely loaded within the timeframe configured through set_page_load_timeout(). But as you have invoked print(self.page_source) the elements rendered within the partially rendered HTML DOM are retrieved.

Now about your individual queries :

• How can I make sure some elements are already loaded? : An ideal testcase would have a definite step e.g. to validate the presence of an element, to validate the visibility of an element or to validate the interactability (while clicking) of element. From this perspective verifying the elements are already loaded may not include the desired element. Hence instead of such a broader search criteria you need to narrow down your search criteria to some thing definite e.g.

  • Page Title
  • Page Heading
  • Presence of an Alert
  • Attribute of an element
  • Presence of an element
  • Presence of a group of elements
  • Visibility of an element
  • Visibility of a group of elements
  • Clickablity of an element
  • StalenessOf of an element
  • FrameToBeAvailableAndSwitchToIt

Implementing these narrowed down search criteria can save a lot of execution time with the help of WebDriverWait inconjunction with expected_conditions.

• How can I make sure all elements are loaded? : Again, our tests should be focused only on the element/elements with which we need to interact with and leaveout verifying the status/condition of other elements which are not of our interest.

• Now, following the two points mentioned above these are the 3 most commonly used usecases :

  • presence_of_element_located : An expectation for checking that an element is present on the DOM of a page. This does not necessarily mean that the element is visible.
  • visibility_of_element_located : An expectation for checking that an element is present on the DOM of a page and visible. Visibility means that the element is not only displayed but also has a height and width that is greater than 0.
  • element_to_be_clickable : An Expectation for checking an element is visible and enabled such that you can click it.

• As per the usecase you mentioned you can make a List of all the <th> elements visible within the DOM Tree whilst waiting for a configurable amount of time inducing WebDriverWait as follows :

  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support import expected_conditions as EC 
  
  headerList = WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//table[@class='fth1_' and @id='fth1_']/thead/tr//th")))
  

Note : The xpath used in this illustration is a sample xpath used only for demonstration purpose.