web scraping - Using Requests and BeautifulSoup - Python returns tag with no text

Question

I'm trying to capture the number of visits on this page, but python returns the tag with no text.

This is what I've done.

import requests
from bs4 import BeautifulSoup

r = requests.get("http://www.kijiji.ca/v-2-bedroom-apartments-condos/city-of-halifax/clayton-park-west-condo-style-luxury-2-bed-den/1016364514")
soup = BeautifulSoup(r.content)
print soup.find_all("span",{"class":"ad-visits"})

Answer 1

The values you are trying to scrape are populated by javascript so beautfulsoup or requests aren't going to work in this case.

You'll need to use something like selenium to get the output.

from bs4 import BeautifulSoup
from selenium import webdriver

driver = webdriver.Firefox()
driver.get("http://www.kijiji.ca/v-2-bedroom-apartments-condos/city-of-halifax/clayton-park-west-condo-style-luxury-2-bed-den/1016364514")
soup = BeautifulSoup(driver.page_source , 'html.parser')
print soup.find_all("span",{"class":"ad-visits"})

Selenium will return the page source as rendered and you can then use beautifulsoup to get the value

[<span class="ad-visits">385</span>]