python - Running command in parallel in pyspark

Question

I have small snippet of code in pyspark.

# save data frame as temp table 
df.createOrReplaceTempView("my_table")

# save data frame as csv
df.write.format("csv").save("my_csv")

Now the 1st command executes then only 2nd will execute.

Is there a way where I can trigger both these commands in parallel i.e I want both commands to execute at the same time

question from:https://stackoverflow.com/questions/66054223/running-command-in-parallel-in-pyspark

Answer 1

You can use python multiprocessing

from multiprocessing import Process

def create_temp_view(df):
    df.createOrReplaceTempView("my_table")

def write_df(df):
    df.write.mode("overwrite").format("csv").save("my_csv")


p1 = Process(create_temp_view(df))
p1.start()
p2 = Process(write_df(df))
p2.start()
p1.join()
p2.join()