python - Why is __init__ not called after __new__ SOMETIMES

Question

Let me start with this is not a repeat of Why does __init__ not get called if __new__ called with no args. I have tried to carefully construct some sample code for __new__ and __init__ that has no explanation I can find.

The basic parameters:

• There is a base class called NotMine as it comes from another library (I'll disclose at the end, not important here)
• That class has an __init__ method that in turn calls a _parse method
• I need to override the _parse method in subclasses
• which subclass I'm creating is not known until invocation
• I know there are factory design methods but I cannot use them here (More at the end)
• I have tried to make careful use of super to avoid the problems in Python logging: Why is __init__ called twice?
• I know this is also 'kind of' an AbstractBaseMehtod opportunity but that did not help

Anyway, __init__ should be called after __new__ and for every explanation of why SOME samples below don't work I seem to be able to point to other cases that do work and rule out the explanation.

class NotMine(object):

    def __init__(self, *args, **kwargs):
        print "NotMine __init__"
        self._parse()

    def _parse(self):
        print "NotMine _parse"

class ABC(NotMine):
    def __new__(cls,name,*args, **kwargs):
        print "-"*80
        print "Entered through the front door ABC.__new__(%s,%s,*%s,**%s)"%(cls,name,args,kwargs)
        if name == 'AA':
            obj = super(NotMine,ABC).__new__(AA,*args,**kwargs)
            print "Exiting door number 1 with an instance of: %s"%type(obj)
            return obj 
        elif name == 'BB':
            obj = super(NotMine,ABC).__new__(BB,*args,**kwargs)
            print "Exiting door number 2 with an instance of: %s"%type(obj)
            return obj
        else:
            obj = super(NotMine,ABC).__new__(cls,*args,**kwargs)
            print "Exiting door number 3 with an instance of: %s"%type(obj)
            return obj

class AA(ABC):

    def _parse(self):
       print "AA _parse"

class BB(ABC):

    def __init__(self, *args, **kw):
        print "BB_init:*%s, **%s"%(args,kw)        
        super(BB,self).__init__(self,*args,**kw)

    def _parse(self):
        print "BB _parse"

class CCC(AA):

    def _parse(self):
        print "CCCC _parse"


print("########### Starting with ABC always calls __init__ ############")
ABC("AA")            # case 1
ABC("BB")            # case 2
ABC("NOT_AA_OR_BB")  # case 3

print("########### These also all call __init__ ############")
AA("AA")           # case 4
BB("BB")           # case 5
AA("NOT_AA_OR_BB") # case 6
BB("NOT_AA_OR_BB") # case 7
CCC("ANYTHING")    # case 8

print("########### WHY DO THESE NOT CALL __init__ ############")
AA("BB")  # case 9  
BB("AA")  # case 10
CCC("BB") # case 11

If you execute the code, you can see that for each call to __new__ it announces "which door" it is exiting through and with what type. I can exit the same "door" with the same "type" object and have __init__ called in one case and not the other. I've looked at the mro of the "calling" class and that offers no insight since I can invoke that class ( or a subcass as in CCC ) and have __init__ called.

End Notes: The NotMine library I'm using is the Genshi MarkupTemplate and the reason for not using a Factory design method is that their TemplateLoader needs a defaultClass to construct. I don't know until I start parsing, which I do in __new__. There is a lot of cool voodoo magic that genshi loaders and templates do that make this worth the effort.

I can run an unmodified instance of their loader and currently everything works as long as I ONLY pass the ABC (abstract sort-of-factory) class as the default. Things are working well but this unexplained behavior is an almost certain bug later.

UPDATE: Ignacio, nailed the top line question, if the returned object is not an "instance of" cls then __init__ is not called. I do find that calling the "constructor" (e.g. AA(args..) is wrong as it will call __new__ again and you are right back where you started. You could modify an arg to take a different path. That just means you call ABC.__new__ twice rather than infinitely. A working solution is to edit class ABC above as:

class ABC(NotMine):
  def __new__(cls,name,*args, **kwargs):
    print "-"*80
    print "Entered through the front door ABC.__new__(%s,%s,*%s,**%s)"%(cls,name,args,kwargs)
    if name == 'AA':
        obj = super(NotMine,ABC).__new__(AA,*args,**kwargs)
        print "Exiting door number 1 with an instance of: %s"%type(obj)
    elif name == 'BB':
        obj = super(NotMine,ABC).__new__(BB,*args,**kwargs)
        print "Exiting door number 2 with an instance of: %s"%type(obj)
    elif name == 'CCC':
        obj = super(NotMine,ABC).__new__(CCC,*args,**kwargs)
        print "Exiting door number 3 with an instance of: %s"%type(obj)
    else:
        obj = super(NotMine,ABC).__new__(cls,*args,**kwargs)
        print "Exiting door number 4 with an instance of: %s"%type(obj)
    ## Addition to decide who calls __init__  ##
    if isinstance(obj,cls):
        print "this IS an instance of %s So call your own dam __init__"%cls
        return obj
    print "this is NOT an instance of %s So __new__ will call __init__ for you"%cls
    obj.__init__(name,*args, **kwargs)
    return obj

print("########### now, these DO CALL __init__ ############")
AA("BB")  # case 9  
BB("AA")  # case 10
CCC("BB") # case 11

Notice the last few lines. Not calling __init__ if it's a "different" class does not make sense to me, ESPECIALLY when the "different" class is still a subclass of the class calling __init__. I don't like the above edit but least I get the rules a little better now.

Answer 1

From the documentation:

If __new__() does not return an instance of cls, then the new instance’s __init__() method will not be invoked.

This is to allow __new__() to return a new instance of a different class, which has its own __init__() to be called instead. You will need to detect if you're creating a new cls, and call the appropriate constructor instead if not.