I have the following code (simplified for clarity):
import os
import errno
import imp
lib_dir = os.path.expanduser('~/.brian/cython_extensions')
module_name = '_cython_magic_5'
module_path = os.path.join(lib_dir, module_name + '.so')
code = 'some code'
have_module = os.path.isfile(module_path)
if not have_module:
pyx_file = os.path.join(lib_dir, module_name + '.pyx')
# THIS IS WHERE EACH PROCESS TRIES TO WRITE TO THE FILE. THE CODE HERE
# PREVENTS A RACE CONDITION.
try:
fd = os.open(pyx_file, os.O_CREAT | os.O_EXCL | os.O_WRONLY)
except OSError as e:
if e.errno == errno.EEXIST:
pass
else:
raise
else:
os.fdopen(fd, 'w').write(code)
# THIS IS WHERE EACH PROCESS TRIES TO READ FROM THE FILE. CURRENTLY THERE IS A
# RACE CONDITION.
module = imp.load_dynamic(module_name, module_path)
(Some of the above code is borrowed from this answer.)
When several processes are run at once, this code causes just one to open and write to pyx_file
(assuming pyx_file
does not already exist). The problem is that as this process is writing to pyx_file
, the other processes try to load pyx_file
-- errors are raised in the latter processes, because at the time they read pyx_file
, it's incomplete. (Specifically, ImportError
s are raised, because the processes are trying to import the contents of the file.)
What's the best way to avoid these errors? One idea is to have the processes keep trying to import pyx_file
in a while loop until the import is successful. (This solution seems suboptimal.)
See Question&Answers more detail:
os 与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…