python - How to create a new columns of dataframe based on string containing condition

Question

I have a pandas dataframe like follows:

df = pd.DataFrame({'Date':['10/2/2011', '11/2/2011', '12/2/2011', '13/2/2011'], 
                    'Phrases':['I have a cool family', 'I like avocados', 'I would like to go to school', 'I enjoy Harry Potter']}) 

and a list of keywords as follows

l=['cool','avocado','lord of the rings']

I want to create a new column in the dataframe with True/False values. It will depend on whether or not each entity in "Phrases" contains one or more keywords for the list "l". In this case, the new column should read True, True, False, False.

This is simple for short dataframes, with

for i in ...
    str(bool([ele for ele in (keyword list) if(ele in df.Phrases[i])] ))

but a for loop is not reasonable for data frames of >1000000 rows, like my real one. Is there a more efficient way to create a new column with these True/False values.

question from:https://stackoverflow.com/questions/65948727/how-to-create-a-new-columns-of-dataframe-based-on-string-containing-condition

Answer 1

You can do it with pd.Series.str.contains with giving the list l as a OR string :

import re
import pandas as pd

df = pd.DataFrame({'Date':['10/2/2011', '11/2/2011', '12/2/2011', '13/2/2011'], 
                    'Phrases':['I have a cool family', 'I like avocados', 'I would like to go to school', 'I enjoy Harry Potter']})

l=['cool','avocado','lord of the rings']

df['new_column']=df['Phrases'].str.contains('|'.join(l))

df['matched strings']=df['Phrases'].apply(lambda x: ','.join(re.findall('|'.join(l),x)))


df
Out[18]: 
        Date                       Phrases  new_column matched strings
0  10/2/2011          I have a cool family        True            cool
1  11/2/2011               I like avocados        True         avocado
2  12/2/2011  I would like to go to school       False                
3  13/2/2011          I enjoy Harry Potter       False