web scraping - Python Twitter: Check if a tweet exists by url without API

Question

I have a list of tweets urls from the same account, and I want to check if this tweets still exist or not.

A tweet may not exist anymore if twitter responds with such errors:

This Tweet is from an account that no longer exists. Learn more

or

Sorry that page doesn't exist!

or any such errors.

What I have tried is using twint library to scrape all the tweets from the given profile, and check if the tweets on my "tweets list" is also in the result that the twint library.

And I have used this function to scrape all the tweets using twint:

def get_tweets(username):
    c = twint.Config()
    c.Username = username
    tweets = []
    c.Store_object = True
    c.Retweets = True
    c.Store_object_tweets_list = tweets
    c.Hide_output = True
    twint.run.Profile(c)
    tweets_links = []
    for tweet in tweets:
        tweets_links.append(tweet.link)
    return tweets_links
get_tweets(username)

This works well but the problem is that it doesn't scrape all the tweets, and it stops at a certain date (for the username I'm testing 'GideonCRozner' it stops at 24/06/2020), and I have posts urls which are before that date. So simply I'm not able to scrape all the posts using twint library.

My solution right now is to include selenium in the code and get the posts which are not scraped yet one by one, but as you know selenium is a slower solution for that.

So I hope that I can use some ideas from you, to scrape all the user's tweets or test a tweet if it exists without selenium and without Twitter API

Thanks a lot for your time!

question from:https://stackoverflow.com/questions/65849991/python-twitter-check-if-a-tweet-exists-by-url-without-api

Answer 1

Waitting for answers