How can I pickle a dynamically created nested class in python?

Question

I have a nested class:

class WidgetType(object):
    
    class FloatType(object):
        pass
    
    class TextType(object):
        pass

.. and an object that refers the nested class type (not an instance of it) like this

class ObjectToPickle(object):
     def __init__(self):
         self.type = WidgetType.TextType

Trying to serialize an instance of the ObjectToPickle class results in:

PicklingError: Can't pickle <class 'setmanager.app.site.widget_data_types.TextType'>

Is there a way to pickle nested classes in python?

Answer 1

I know this is a very old question, but I have never explicitly seen a satisfactory solution to this question other than the obvious, and most likely correct, answer to re-structure your code.

Unfortunately, it is not always practical to do such a thing, in which case as a very last resort, it is possible to pickle instances of classes which are defined inside another class.

The python documentation for the __reduce__ function states that you can return

A callable object that will be called to create the initial version of the object. The next element of the tuple will provide arguments for this callable.

Therefore, all you need is an object which can return an instance of the appropriate class. This class must itself be picklable (hence, must live on the __main__ level), and could be as simple as:

class _NestedClassGetter(object):
    """
    When called with the containing class as the first argument, 
    and the name of the nested class as the second argument,
    returns an instance of the nested class.
    """
    def __call__(self, containing_class, class_name):
        nested_class = getattr(containing_class, class_name)
        # return an instance of a nested_class. Some more intelligence could be
        # applied for class construction if necessary.
        return nested_class()

All that is left therefore, is to return the appropriate arguments in a __reduce__ method on FloatType:

class WidgetType(object):

    class FloatType(object):
        def __reduce__(self):
            # return a class which can return this class when called with the 
            # appropriate tuple of arguments
            return (_NestedClassGetter(), (WidgetType, self.__class__.__name__, ))

The result is a class which is nested but instances can be pickled (further work is needed to dump/load the __state__ information, but this is relatively straightforward as per the __reduce__ documentation).

This same technique (with slight code modifications) can be applied for deeply nested classes.

A fully worked example:

import pickle


class ParentClass(object):

    class NestedClass(object):
        def __init__(self, var1):
            self.var1 = var1

        def __reduce__(self):
            state = self.__dict__.copy()
            return (_NestedClassGetter(), 
                    (ParentClass, self.__class__.__name__, ), 
                    state,
                    )


class _NestedClassGetter(object):
    """
    When called with the containing class as the first argument, 
    and the name of the nested class as the second argument,
    returns an instance of the nested class.
    """
    def __call__(self, containing_class, class_name):
        nested_class = getattr(containing_class, class_name)

        # make an instance of a simple object (this one will do), for which we can change the
        # __class__ later on.
        nested_instance = _NestedClassGetter()

        # set the class of the instance, the __init__ will never be called on the class
        # but the original state will be set later on by pickle.
        nested_instance.__class__ = nested_class
        return nested_instance



if __name__ == '__main__':

    orig = ParentClass.NestedClass(var1=['hello', 'world'])

    pickle.dump(orig, open('simple.pickle', 'w'))

    pickled = pickle.load(open('simple.pickle', 'r'))

    print type(pickled)
    print pickled.var1

My final note on this is to remember what the other answers have said:

If you are in a position to do so, consider re-factoring your code to avoid the nested classes in the first place.