python - line 105, in spawn_main exitcode = _main(fd) while multiprocessing in a for loop

Question

I had a look into many published issues without finding some insights to my current issue.

I am dealing with multiprocessing runs of an external code. This external code eats inputs files. The files names are joined in a list that enable me to launch pool for each file. A path is also needed.

for i in range(len(file2run)):
    pool.apply_async(runcase, args=(file2run[i], filepath))

The runcase function launches one process for a given input file and analyses and saves the results in some folder.

it works fine whatever the length of the file2run is. The external code runs on several processes (as many as maxCPU : defined in the pool with:

pool = multiprocessing.Pool(processes = maxCPU).  

My issue is that I'd like to make a step further and integrate this in a for loop. In each loop, several input files are created and once all of the runs are finished a new set of inputs files are created and a pool is created again. It works fine for two loops but I encountered the issue of the xxx line 105, in spawn_main exitcode = _main(fd) and a bunch of messages up the error of a missing needed module. Same messages for 2 or 1000 input files in each loop... So I guess it's about the pool creation, but is there a way of clearing the variables between each runs ?? I have tried to created the pool initialization (with the number of CPU) at the very beginning of the main function but same issues raises...I have tried to make a sort of equivalent of clear all matlab function but always same issue... and why does it work for two loops and not for the third one ? why is the 2nd one working??

Thanks in advance for any help (or to point out to the good already published issue).

Xavfa

Answer 1

here is a try of an example that actually......works ! I copypaste my original script and made it way more easier to share for sake of understanding the paradigm of my original try (the original one deals with object of several kinds to build the input file and uses an embedded function of one of the objects to launches the external code with subprocess.check_all). but the example keeps the over all paradigm of making input files in a folder, simulation results in an other one with multiprocessing package. the original still doesn't work, still at the third round of the loop (if name == 'main' : of multiproc_test.py

here is one script (multiproc_test.py):

import os
import Simlauncher

def RunProcess(MainPath):
    file2run = Simlauncher.initiateprocess(MainPath)
    Simlauncher.RunMultiProc(file2run, MainPath, multi=True, maxcpu=0.7)

def LaunchProcess(nbcase):
    #exemple that build the file
    MainPath = os.getcwd()
    SimDir = os.path.join(os.getcwd(), 'SimFiles\')
    if not os.path.exists(SimDir):
        os.mkdir(SimDir)

    for i in range(100):
        with open(SimDir+'inputfile'+str(i)+'.mptest', 'w') as file:
            file.write('Hello World')

    RunProcess(MainPath)

if __name__ == '__main__' :
    for i in range(1,10):
        LaunchProcess(i)
        os.rename(os.path.join(os.getcwd(), 'SimFiles'), os.path.join(os.getcwd(),            'SimFiles'+str(i)))

here is the other one (Simlauncher.py) :

import multiprocessing as mp
import os

def initiateprocess(MainPath):
    filepath = MainPath + '\SimFiles\'
    listOfFiles = os.listdir(filepath)
    file2run = []
    for file in listOfFiles:
        if '.mptest' in file:
            file2run.append(file)
    return file2run

def runtestcase(file,filepath):
    filepath = filepath+'\SimFiles'
    ResSimpath = filepath + '\SimRes\'
    if not os.path.exists(ResSimpath):
        os.mkdir(ResSimpath)
    with open(ResSimpath+'Res_' + file, 'w') as res:
        res.write('I am done')
    print(file +'is finished')

def RunMultiProc(file2run, filepath, multi, maxcpu):
    print('Launching cases :')
    nbcpu = mp.cpu_count()
    pool = mp.Pool(processes=int(nbcpu * maxcpu)) 
    for i in range(len(file2run)):
        pool.apply_async(runtestcase, args=(file2run[i], filepath))
    pool.close()
    pool.join()
    print('Done with this one !')

any help is still needed.... btw, the external code is energyplus (for building energy simulation) Xavier