beautifulsoup - Python Webscraping, how to navigate on a website?

Question

I am programming a program that should read out certain data from a website and only output certain data (data from a table). However, I ran into a problem. I wrote a program that logs into the website, but from that website I have to go to the next website and then open the document with the data. Unfortunately, I have no idea how I can change the website and then open the document and read out the data. Does anyone have any idea how I could get on there?

from bs4 import BeautifulSoup
import requests

User = ''
Pass = ''
LOGIN_URL = ''
LOGIN_API_URL = ''


def main():
    session_requests = requests.session()
    result = session_requests.get(LOGIN_URL)
    cookies = result.cookies
    soup = BeautifulSoup(result.content, "html.parser")
    auth_token = soup.find("input", {'name': 'logintoken'}).get('value')

    payload = {'username': User, 'password': Pass , 'logintoken':auth_token }

    result = session_requests.post(
    LOGIN_API_URL,
    data=payload,
    cookies=cookies
    )

    #Report successful login
    print("Login succeeded: ", result.ok)
    print("Status code:", result.status_code)
    print(result.text)

    #Get Data



    # Close Session
    requests.session().close()
    print('Session closed')

# Entry point
if __name__ == '__main__':
    main()

question from:https://stackoverflow.com/questions/66059096/python-webscraping-how-to-navigate-on-a-website

Answer 1

You should read into Selenium with Python. Since there is no specific URL or login details (which you shouldn't post here anyway) it would be quite hard for any of us to create a working example since we don't have anything to work with.

Try the using selenium from the link above and if you have any questions or run into any issues from there come back and ask that specific question.

BS4 and requests can be powerful but selenium emulates a web browser and lets you move through websites like a "human" would. Start there.