macos - Python multiprocessing with M1 Mac

Question

I have a mac (Mac Os 11.1, Python Ver 3.8.2) and need to work in multiprocessing, but the procedures doesn’t work.

import multiprocessing


def func(index: int):
    print(index)


manager = multiprocessing.Manager()
processes = []

for i in range(-1, 10):
    p = multiprocessing.Process(target=func,
                                args=(i,))
    processes.append(p)
    p.start()

for process in processes:
    process.join()

However, on my Intel-based Mac, it works fine.

What I expect is

-1
0
1
2
3
4
5
6
7
8
9

But instead, I got an error:

RuntimeError: 
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.
Traceback (most recent call last):
  File "/Users/lance/Documents/Carleton Master Projects/Carleton-Master-Thesis/experiment.py", line 7, in <module>
    manager = multiprocessing.Manager()
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 57, in Manager
    m.start()
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/managers.py", line 583, in start
    self._address = reader.recv()
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/connection.py", line 383, in _recv
    raise EOFError
EOFError

Is there any similar way (also keep it easy) to parallelize in M1-based Mac?

question from:https://stackoverflow.com/questions/65859890/python-multiprocessing-with-m1-mac

Answer 1

I'm not sure why this works on an Intel machine, but the problem is all the MP-related code should be inside if __name__ == '__main__':

import multiprocessing

def func(index: int):
    print(index)

if __name__ == '__main__':
    manager = multiprocessing.Manager()
    processes = []

    for i in range(-1, 10):
    p = multiprocessing.Process(target=func, args=(i,))
    processes.append(p)
    p.start()

    for process in processes:
        process.join()

Since MP actually starts a new process, the new process needs to import the func function from this module to execute its task. If you don't protect the code that starts new processes, it will be executed during this import.