Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
409 views
in Technique[技术] by (71.8m points)

python - How to efficiently check if a list of words is contained in a Spark Dataframe?

Using PySpark dataframes I'm trying to do the following as efficiently as possible. I have a dataframe with a column which contains text and a list of words I want to filter rows by. So:

Dataframe would look like this

df:
col1    col2   col_with_text
a       b      foo is tasty
12      34     blah blahhh
yeh     0      bar of yums

The list will be list = [foo,bar] And thus result will be:

result:
col1    col2   col_with_text
a       b      foo
yeh     0      bar

Afterwards not only identical string matching will be done but also tested for similarity by using SequenceMatcher or so. This is what I already tried:

def check_keywords(x):
   words_list = ['foo','bar']

   for word in x
       if word == words_list[0] or word == words_list[1]:
           return x

result = df.map(lambda x: check_keywords(x)).collect()

Unfortunately I was unsuccesfull, could someone help me out? Thanks in advance.

See Question&Answers more detail:os

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Reply

0 votes
by (71.8m points)

You should consider using pyspark sql module functions instead of writing a UDF, there are several regexp based functions:

First let's start with a more complete sample data frame:

df = sc.parallelize([["a","b","foo is tasty"],["12","34","blah blahhh"],["yeh","0","bar of yums"], 
                     ['haha', '1', 'foobar none'], ['hehe', '2', 'something bar else']])
    .toDF(["col1","col2","col_with_text"])

If you want to filter lines based on whether they contain one of the words in words_list, you can use rlike:

import pyspark.sql.functions as psf
words_list = ['foo','bar']
df.filter(psf.col('col_with_text').rlike('(^|s)(' + '|'.join(words_list) + ')(s|$)')).show()

    +----+----+------------------+
    |col1|col2|     col_with_text|
    +----+----+------------------+
    |   a|   b|      foo is tasty|
    | yeh|   0|       bar of yums|
    |hehe|   2|something bar else|
    +----+----+------------------+

If you want to extract the strings matching the regular expression, you can use regexp_extract:

df.withColumn(
        'extracted_word', 
        psf.regexp_extract('col_with_text', '(?=^|s)(' + '|'.join(words_list) + ')(?=s|$)', 0))
    .show()

    +----+----+------------------+--------------+
    |col1|col2|     col_with_text|extracted_word|
    +----+----+------------------+--------------+
    |   a|   b|      foo is tasty|           foo|
    |  12|  34|       blah blahhh|              |
    | yeh|   0|       bar of yums|           bar|
    |haha|   1|       foobar none|              |
    |hehe|   2|something bar else|              |
    +----+----+------------------+--------------+

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
OGeek|极客中国-欢迎来到极客的世界,一个免费开放的程序员编程交流平台!开放,进步,分享!让技术改变生活,让极客改变未来! Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...