Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
926 views
in Technique[技术] by (71.8m points)

python - How to prevent a race condition when multiple processes attempt to write to and then read from a file at the same time

I have the following code (simplified for clarity):

import os
import errno
import imp


lib_dir = os.path.expanduser('~/.brian/cython_extensions')
module_name = '_cython_magic_5'
module_path = os.path.join(lib_dir, module_name + '.so')
code = 'some code'

have_module = os.path.isfile(module_path)
if not have_module:
    pyx_file = os.path.join(lib_dir, module_name + '.pyx')

    # THIS IS WHERE EACH PROCESS TRIES TO WRITE TO THE FILE.  THE CODE HERE 
    # PREVENTS A RACE CONDITION.
    try:
        fd = os.open(pyx_file, os.O_CREAT | os.O_EXCL | os.O_WRONLY)
    except OSError as e:
        if e.errno == errno.EEXIST:
            pass
        else:
            raise
    else:
        os.fdopen(fd, 'w').write(code)

# THIS IS WHERE EACH PROCESS TRIES TO READ FROM THE FILE.  CURRENTLY THERE IS A
# RACE CONDITION.
module = imp.load_dynamic(module_name, module_path)

(Some of the above code is borrowed from this answer.)

When several processes are run at once, this code causes just one to open and write to pyx_file (assuming pyx_file does not already exist). The problem is that as this process is writing to pyx_file, the other processes try to load pyx_file -- errors are raised in the latter processes, because at the time they read pyx_file, it's incomplete. (Specifically, ImportErrors are raised, because the processes are trying to import the contents of the file.)

What's the best way to avoid these errors? One idea is to have the processes keep trying to import pyx_file in a while loop until the import is successful. (This solution seems suboptimal.)

See Question&Answers more detail:os

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Reply

0 votes
by (71.8m points)

The way to do this is to take an exclusive lock each time you open it. The writer holds the lock while writing data, while the reader blocks until the writer releases the lock with the fdclose call. This will of course fail if the file has been partially written and the writing process exits abnormally, so a suitable error to delete the file should be displayed if the module can't be loaded:

import os
import fcntl as F

def load_module():
    pyx_file = os.path.join(lib_dir, module_name + '.pyx')

    try:
        # Try and create/open the file only if it doesn't exist.
        fd = os.open(pyx_file, os.O_CREAT | os.O_EXCL | os.O_WRONLY):

        # Lock the file exclusively to notify other processes we're writing still.
        F.flock(fd, F.LOCK_EX)
        with os.fdopen(fd, 'w') as f:
            f.write(code)

    except OSError as e:
        # If the error wasn't EEXIST we should raise it.
        if e.errno != errno.EEXIST:
            raise

    # The file existed, so let's open it for reading and then try and
    # lock it. This will block on the LOCK_EX above if it's held by
    # the writing process.
    with file(pyx_file, "r") as f:
        F.flock(f, F.LOCK_EX)

    return imp.load_dynamic(module_name, module_path)

module = load_module()

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
OGeek|极客中国-欢迎来到极客的世界,一个免费开放的程序员编程交流平台!开放,进步,分享!让技术改变生活,让极客改变未来! Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...