multithreading - python 2.6.x theading / signals /atexit fail on some versions?

Question

I've seen a lot of questions related to this... but my code works on python 2.6.2 and fails to work on python 2.6.5. Am I wrong in thinking that the whole atexit "functions registered via this module are not called when the program is killed by a signal" thing shouldn't count here because I'm catching the signal and then exiting cleanly? What's going on here? Whats the proper way to do this?

import atexit, sys, signal, time, threading

terminate = False
threads = []

def test_loop():
    while True:
        if terminate:
            print('stopping thread')
            break
        else:
            print('looping')
            time.sleep(1)

@atexit.register

def shutdown():
    global terminate
    print('shutdown detected')
    terminate = True
    for thread in threads:
        thread.join()

def close_handler(signum, frame):
    print('caught signal')
    sys.exit(0)

def run():
    global threads
    thread = threading.Thread(target=test_loop)
    thread.start()
    threads.append(thread)

    while True:
        time.sleep(2)
        print('main')

signal.signal(signal.SIGINT, close_handler)

if __name__ == "__main__":
    run()

python 2.6.2:

$ python halp.py 
looping
looping
looping
main
looping
main
looping
looping
looping
main
looping
^Ccaught signal
shutdown detected
stopping thread

python 2.6.5:

$ python halp.py 
looping
looping
looping
main
looping
looping
main
looping
looping
main
^Ccaught signal
looping
looping
looping
looping
...
looping
looping
Killed <- kill -9 process at this point

The main thread on 2.6.5 appears to never execute the atexit functions.

Answer 1

The root difference here is actually unrelated to both signals and atexit, but rather a change in the behavior of sys.exit.

Before around 2.6.5, sys.exit (more accurately, SystemExit being caught at the top level) would cause the interpreter to exit; if threads were still running, they'd be terminated, just as with POSIX threads.

Around 2.6.5, the behavior changed: the effect of sys.exit is now essentially the same as returning from the main function of the program. When you do that--in both versions--the interpreter waits for all threads to be joined before exiting.

The relevant change is that Py_Finalize now calls wait_for_thread_shutdown() near the top, where it didn't before.

This behavioral change seems incorrect, primarily because it no longer functions as documented, which is simply: "Exit from Python." The practical effect is no longer to exit from Python, but simply to exit the thread. (As a side note, sys.exit has never exited Python when called from another thread, but that obscure divergance from documented behavior doesn't justify a much bigger one.)

I can see the appeal of the new behavior: rather than two ways to exit the main thread ("exit and wait for threads" and "exit immediately"), there's only one, as sys.exit is essentially identical to simply returning from the top function. However, it's a breaking change and diverges from documented behavior, which far outweighs that.

Because of this change, after sys.exit from the signal handler above, the interpreter sits around waiting for threads to exit and then runs atexit handlers after they do. Since it's the handler itself that tells the threads to exit, the result is a deadlock.