python - Python3's super and comprehensions -> TypeError?

Question

Using python3's super in a comprehension seems to always result in TypeError: super(type, obj): obj must be an instance or subtype of type (but using python 2's super does work as expected)

class A(object):
    def __repr__(self):
         return "hi!"  

class B(A):
    def __repr__(self):
         return "".join(super().__repr__() for i in range(2))  

repr(B())
#output: <repr(<__main__.B at 0x7f70cf36fcc0>) failed: TypeError: super(type, obj): obj must be an instance or subtype of type>

class C(A):
    def __repr__(self):
        s = ''
        for i in range(4):
            s += super().__repr__()
        return s     

repr(C())
#output: hi!hi!hi!hi!

class D(A):
    def __repr__(self):
        return "".join(super(D,self).__repr__() for i in range(4))

repr(D())
#output: hi!hi!hi!hi!

So, why does new super() fail in generator comprehensions?

Addendum:

In [28]: class E(A):
   ....:     def __repr__(self):
   ....:         def inner():
   ....:             print(repr(__class__))
   ....:         inner()
   ....:         return ''
In [29]: repr(E())
<class '__main__.E'>
Out[29]: ''

In [30]: class F(A):
   ....:     def __repr__(self):
   ....:         return "".join([super().__repr__() for i in range(4)])
   ....:     

In [31]: repr(F())

TypeError: super(type, obj): obj must be an instance or subtype of type

Answer 1

Simple explanation

Look at the documentation for super():

The zero argument form only works inside a class definition, as the compiler fills in the necessary details to correctly retrieve the class being defined, as well as accessing the current instance for ordinary methods.

By inside a class definition they meant inside class method scope. Inside class method scope interpreter is able to complete zero form with same parameters as you would explicitly provide it in Python 2. List comprehension however creates it's own scope. That's the reason why it fails: you call to super() not from the class method scope and interpreter is not able to complete it with all parameters.

Advanced explanation

According to Python data model:

__class__ is an implicit closure reference created by the compiler if any methods in a class body refer to either __class__ or super. This allows the zero argument form of super() to correctly identify the class being defined based on lexical scoping, while the class or instance that was used to make the current call is identified based on the first argument passed to the method.

Python is able to collect first parameter for super() from __class__ variable even inside list comprehension (since it is available in all child scopes as any usual closure). You can test it using:

class T:
    def test(self):
        print(__class__)
        print([__class__ for _ in range(1)][0])


T().test()

Will output:

<class '__main__.T'>
<class '__main__.T'>

But interpreter incorrectly collects second parameter for super(): self. It assumes that call to super() happens inside method scope and tries to get first parameter for the method using following C code (many lines are omitted for clarity):

PyFrameObject *f = PyThreadState_GET()->frame;
obj = f->f_localsplus[0];
if (obj != NULL) {
    obj_type = supercheck(type, obj);
    if (obj_type == NULL)
        return -1;
    Py_INCREF(obj);
}

It's not possible to access f->f_localsplus[0] from Python, but it contains "locals+stack" according to comment in code. So we can utilize locals() for a test (unfortunately order is missing). Let's test, what is available in locals inside class method and list comprehension:

class T:
    def test(self):
        print(locals())
        print([locals() for _ in range(1)])


T().test()

Will print:

{'self': <__main__.T object at 0x100f1f8d0>}
{'_': 0, '.0': <range_iterator object at 0x100fbb2a0>}

There is a reference to our object in the first case and it will be correctly found by interpreter. Inside list comprehension there is no object inside dictionary so it will get either 0 or range_iterator (remember, order is missing?). Neither of those is instance of our object. It will fail supercheck() and give you an error obj must be an instance or subtype of type (e.g. T).

---

Take a look here for more information on how super() is implemented and here for more details why it's done like this.