time - Stop processes from python multiprocessing that take too long

Question

I want to evaluate a very complicated (and possibly slow) function for N different input values.

def f(x):
   # do something very complicated
   return out

Via multiprocessing I want to parallelize the calculation on a cluster with N cores, so I do

import multiprocessing  
pool = Pool(N)
results = pool.map(f, xlist)
pool.close()
pool.join()

with 'xlist' being a list (or numpy array) that contains all input values. My question is the following: There might be values of x, where the solution of f does not converge, but I don't want to occupy the whole cluster, just because one data point does not finish. So how can I time each process, and if the time's up, return None, or some different placeholder instead of the usual output of f?

question from:https://stackoverflow.com/questions/65928485/stop-processes-from-python-multiprocessing-that-take-too-long

Answer 1

Waitting for answers