How to increase a function speed being calling 400 time in Python

Question

I have a list named dfs. It contains 400 Pandas dataframes of size 700 rows x 400 columns.

I have a function like this:

def updateDataframe(i):
    global dfs
    df = dfs[i]

    df["abc"].iloc[-1] = "xyz"

    df["abc2"] = df["abc"].rolling(10).mean()

    ........ #More pandas operations like this

    dfs[i] = df



for i in range(len(dfs)):
    updateDataframe(i)

Now, this loop takes 10 seconds to execute. I have tried python multi-processing, but it takes same time and somtimes even more.

Things I tried:

import multiprocessing.dummy as mp #Multi process Library, used for speeding up download
p=mp.Pool(8) #Define Number of Process to Use
p.map(updateDataframe,range(len(dfs))) # Call the Download Image funciton
p.close() #Close the multi threads
p.join()

Also tried this:

from multiprocessing import Process

if __name__ == "__main__":  # confirms that the code is under main function
    processes = []
    for i in range(len(dfs)):
        process = Process(target=updateDataframe, args=(i,))
        processes.append(process)
        processes.start()

    # complete the processes
    for i in range(len(processes)):
        processes[i].join()

question from:https://stackoverflow.com/questions/65932578/how-to-increase-a-function-speed-being-calling-400-time-in-python

Answer 1

Waitting for answers