closures - Python: LOAD_FAST vs. LOAD_DEREF with inplace addition

Question

Last Friday I went to a job interview and had to answer the following question: why does this code raise an exception (UnboundLocalError: local variable 'var' referenced before assignment on the line containing var += 1)?

def outer():
    var = 1

    def inner():
        var += 1
        return var

    return inner

I couldn't give a proper answer; this fact really upset me, and when I came home I tried really hard to find a proper answer. Well, I have found the answer, but now there's something else that confuses me.

I have to say in advance that my question is more about the decisions made when designing the language, not about how it works.

So, consider this code. The inner function is a python closure, and var is not local for outer - it is stored in a cell (and then retrieved from a cell):

def outer():
    var = 1

    def inner():
        return var

    return inner

The disassembly looks like this:

0  LOAD_CONST               1 (1)
3  STORE_DEREF              0 (var)  # not STORE_FAST

6  LOAD_CLOSURE             0 (var)
9  BUILD_TUPLE              1
12 LOAD_CONST               2 (<code object inner at 0x10796c810)
15 LOAD_CONST               3 ('outer.<locals>.inner')
18 MAKE_CLOSURE             0
21 STORE_FAST               0 (inner)

24 LOAD_FAST                0 (inner)
27 RETURN_VALUE

recursing into <code object inner at 0x10796c810:

0  LOAD_DEREF               0 (var)  # same thing
3  RETURN_VALUE

This changes when we try to bind something else to var inside the inner function:

def outer():
    var = 1

    def inner():
        var = 2
        return var

    return inner

Yet again the disassembly:

0  LOAD_CONST               1 (1)
3  STORE_FAST               0 (var)  # this one changed
6  LOAD_CONST               2 (<code object inner at 0x1084a1810)
9  LOAD_CONST               3 ('outer.<locals>.inner')
12 MAKE_FUNCTION            0  # AND not MAKE_CLOSURE
15 STORE_FAST               1 (inner)

18 LOAD_FAST                1 (inner)
21 RETURN_VALUE

recursing into <code object inner at 0x1084a1810:

0  LOAD_CONST               1 (2)
3  STORE_FAST               0 (var)  # 'var' is supposed to be local

6  LOAD_FAST                0 (var)  
9  RETURN_VALUE

We store var locally, which complies to what is said in the documentation: assignments to names always go into the innermost scope.

Now, when we try to make an increment var += 1, a nasty LOAD_FAST shows up, which tries to get var from inner's local scope:

14 LOAD_FAST                0 (var)
17 LOAD_CONST               2 (2)
20 INPLACE_ADD
21 STORE_FAST               0 (var)

And of course we get an error. Now, here is what I don't get: why can't we retrieve var with a LOAD_DEREF, and THEN store it inside inner's scope with a STORE_FAST? I mean, this seems to be O.K. with the "innermost scope" assignment stuff, and in the same time it's somewhat more intuitively desirable. At least the += code would do what we want it to do, and I can't come up with a situation in which the described approach could mess something up.

Can you? I feel that I'm missing something here.

Answer 1

Python has a very simple rule that assigns each name in a scope to exactly one category: local, enclosing, or global/builtin.

(CPython, of course, implements that rule by using FAST locals, DEREF closure cells, and NAME or GLOBAL lookups.)

---

Your changed rule does make sense for your dead-simple case, but it's easy to come up with cases where it would be ambiguous (at least for a human reader, if not for the compiler). For example:

def outer():
    var = 1

    def inner():
        if spam:
            var = 1
        var += 1
        return var

    return inner

Does that var += 1 do a LOAD_DEREF or LOAD_FAST? We can't know until we know the value of spam at runtime. Which means we can't compile the function body.

---

Even if you could come up with a more complicated rule that makes sense, there's virtue inherent in the rule being simple. Besides being easy to implement (and therefore easy to debug, optimize, etc.), it's easy for someone to understand. When you get an UnboundLocalError, any intermediate-level Python programmer knows how to work through the rule in his head and figure out what went wrong.

---

Meanwhile, notice that when this comes up in real-life code, there are very easy ways to work around it explicitly. For example:

def inner():
    lvar = var + 1
    return lvar

You wanted to load the closure variable, and assign to a local variable. There's no reason they need to have the same name. In fact, using the same name is misleading, even with your new rule—it implies to the reader that you're modifying the closure variable, when you really aren't. So just give them different names, and the problem goes away.

And that still works with the nonlocal assignment:

def inner():
    nonlocal var
    if spam:
        var = 1
    lvar = var + 1
    return lvar

Or, of course, there are tricks like using a parameter default value to create a local that starts off with a copy of the closure variable:

def inner(var=var):
    var += 1
    return var