To Metaclass or Not to Metaclass

Most of this book has focused on straightforward application-coding techniques—the modules, functions, and classes that most programmers spend their time writing to achieve real-world goals. The majority of Python’s users may use classes and make instances, and might even do a bit of operator overloading, but they probably won’t get too deep into the details of how their classes actually work.

However, in this book we’ve also seen a variety of tools that allow us to control Python’s behavior in generic ways, and that often have more to do with Python internals or tool building than with application-programming domains. As a review, and to help us place metaclasses in the tools spectrum:

As mentioned in this chapter’s introduction, metaclasses are a continuation of this story—they allow us to insert logic to be run automatically at the end of a class statement, when a class object is being created. Though strongly reminiscent of class decorators, the metaclass mechanism doesn’t rebind the class name to a decorator callable’s result, but rather routes creation of the class itself to specialized logic.

In other words, metaclasses are ultimately just another way to define automatically run code. With the tools listed in the prior section, Python provides ways for us to interject logic in a variety of contexts—at operator evaluation, attribute access, function calls, class instance creation, and now class object creation. It’s a language with hooks galore—a feature open to abuse like any other, but one that also offers the flexibility that some programmers desire, and that some programs may require.

As we’ve also seen, many of these advanced Python tools have intersecting roles. For example, attributes can often be managed with properties, descriptors, or attribute interception methods. As we’ll see in this chapter, class decorators and metaclasses can often be used interchangeably as well. By way of preview:

In fact, the main functional difference between these two tools is simply their place in the timing of class creation. As we saw in the prior chapter, class decorators run after the decorated class has already been created. Thus, they are often used to add logic to be run at instance creation time. When they do provide behavior for a class, it is typically through changes or proxies, instead of a more direct relationship.

As we’ll see here, metaclasses, by contrast, run during class creation to make and return the new client class. Therefore, they are often used for managing or augmenting classes themselves, and can even provide methods to process the classes that are created from them, via a direct instance relationship.

For example, metaclasses can be used to add decoration to all methods of classes automatically, register all classes in use to an API, add user-interface logic to classes automatically, create or extend classes from simplified specifications in text files, and so on. Because they can control how classes are made—and by proxy the behavior their instances acquire—metaclass applicability is potentially very wide.

As we’ll also see here, though, these two tools are more similar than different in many common roles. Since tool choices are sometimes partly subjective, knowledge of the alternatives can help you pick the right tool for a given task. To understand the options better, let’s see how metaclasses stack up.

Also like the decorators of the prior chapter, metaclasses are often optional from a theoretical perspective. We can usually achieve the same effect by passing class objects through manager functions—sometimes known as helper functions—much as we can achieve the goals of decorators by passing functions and instances through manager code. Just like decorators, though, metaclasses:

To illustrate, suppose we want to automatically insert a method into a set of classes. Of course, we could do this with simple inheritance, if the subject method is known when we code the classes. In that case, we can simply code the method in a superclass and have all the classes in question inherit from it:

class Extras:
    def extra(self, args):              # Normal inheritance: too static
        ...

class Client1(Extras): ...              # Clients inherit extra methods
class Client2(Extras): ...
class Client3(Extras): ...

X = Client1()                           # Make an instance
X.extra()                               # Run the extra methods

Sometimes, though, it’s impossible to predict such augmentation when classes are coded. Consider the case where classes are augmented in response to choices made in a user interface at runtime, or to specifications typed in a configuration file. Although we could code every class in our imaginary set to manually check these, too, it’s a lot to ask of clients (required is abstract here—it’s something to be filled in):

def extra(self, arg): ...

class Client1: ...                      # Client augments: too distributed
if required():
    Client1.extra = extra

class Client2: ...
if required():
    Client2.extra = extra

class Client3: ...
if required():
    Client3.extra = extra

X = Client1()
X.extra()

We can add methods to a class after the class statement like this because a class-level method is just a function that is associated with a class and has a first argument to receive the self instance. Although this works, it might become untenable for larger method sets, and puts all the burden of augmentation on client classes (and assumes they’ll remember to do this at all!).

It would be better from a maintenance perspective to isolate the choice logic in a single place. We might encapsulate some of this extra work by routing classes through a manager function—such a manager function would extend the class as required and handle all the work of runtime testing and configuration:

def extra(self, arg): ...

def extras(Class):                      # Manager function: too manual
    if required():
        Class.extra = extra

class Client1: ...
extras(Client1)

class Client2: ...
extras(Client2)

class Client3: ...
extras(Client3)

X = Client1()
X.extra()

This code runs the class through a manager function immediately after it is created. Although manager functions like this one can achieve our goal here, they still put a fairly heavy burden on class coders, who must understand the requirements and adhere to them in their code. It would be better if there was a simple way to enforce the augmentation in the subject classes, so that they don’t need to deal with the augmentation so explicitly, and would be less likely to forget to use it altogether. In other words, we’d like to be able to insert some code to run automatically at the end of a class statement, to augment the class.

This is exactly what metaclasses do—by declaring a metaclass, we tell Python to route the creation of the class object to another class we provide:

def extra(self, arg): ...

class Extras(type):
    def __init__(Class, classname, superclasses, attributedict):
        if required():
            Class.extra = extra

class Client1(metaclass=Extras): ...    # Metaclass declaration only (3.X form)
class Client2(metaclass=Extras): ...    # Client class is instance of meta
class Client3(metaclass=Extras): ...

X = Client1()                           # X is instance of Client1
X.extra()

Because Python invokes the metaclass automatically at the end of the class statement when the new class is created, it can augment, register, or otherwise manage the class as needed. Moreover, the only requirement for the client classes is that they declare the metaclass; every class that does so will automatically acquire whatever augmentation the metaclass provides, both now and in the future if the metaclass changes.

Of course, this is the standard rationale, which you’ll need to judge for yourself—in truth, clients might forget to list a metaclass just as easily as they could forget to call a manager function! Still, the explicit nature of metaclasses may make this less likely. Moreover, metaclasses have additional potentials we haven’t yet seen. Although it may be difficult to glean from this small example, metaclasses generally handle such tasks better than more manual approaches.

Having said that, it’s also important to note that the class decorators described in the preceding chapter sometimes overlap with metaclasses—in terms of both utility and benefit. Although they are often used for managing instances, class decorators can also augment classes, independent of any created instances. Their syntax makes their usage similarly explicit, and arguably more obvious than manager function calls.

For example, suppose we coded our manager function to return the augmented class, instead of simply modifying it in place. This would allow a greater degree of flexibility, because the manager would be free to return any type of object that implements the class’s expected interface:

def extra(self, arg): ...

def extras(Class):
    if required():
        Class.extra = extra
    return Class

class Client1: ...
Client1 = extras(Client1)

class Client2: ...
Client2 = extras(Client2)

class Client3: ...
Client3 = extras(Client3)

X = Client1()
X.extra()

If you think this is starting to look reminiscent of class decorators, you’re right. In the prior chapter we emphasized class decorators’ role in augmenting instance creation calls. Because they work by automatically rebinding a class name to the result of a function, though, there’s no reason that we can’t use them to augment the class by changing it before any instances are ever created. That is, class decorators can apply extra logic to classes, not just instances, at class creation time:

def extra(self, arg): ...

def extras(Class):
    if required():
        Class.extra = extra
    return Class

@extras
class Client1: ...             # Client1 = extras(Client1)

@extras
class Client2: ...             # Rebinds class independent of instances

@extras
class Client3: ...

X = Client1()                  # Makes instance of augmented class
X.extra()                      # X is instance of original Client1

Decorators essentially automate the prior example’s manual name rebinding here. Just as for metaclasses, because this decorator returns the original class, instances are made from it, not from a wrapper object. In fact, instance creation is not intercepted at all in this example.

In this specific case—adding methods to a class when it’s created—the choice between metaclasses and decorators is somewhat arbitrary. Decorators can be used to manage both instances and classes, and intersect most strongly with metaclasses in the second of these roles, but this discrimination is not absolute. In fact, the roles of each are determined in part by their mechanics.

As we’ll see ahead, decorators technically correspond to metaclass __init__ methods, used to initialize newly created classes. Metaclasses have additional customization hooks beyond class initialization, though, and may perform arbitrary class construction tasks that might be more difficult with decorators. This can make them more complex, but also better suited for augmenting classes as they are being formed.

For example, metaclasses also have a __new__ method used to create a class, which has no analogy in decorators; making a new class in a decorator would incur an extra step. Moreover, metaclasses may also provide behavior acquired by classes in the form of methods, which have no direct counterpart in decorators either; decorators must provide class behavior is less direct ways.

Conversely, because metaclasses are designed to manage classes, applying them to managing instances alone is less optimal. Because they are also responsible for making the class itself, metaclasses incur this as an extra step in instance management roles.

We’ll explore these differences in code later in this chapter, and will flesh out this section’s partial code into a real working example later in this chapter. To understand how metaclasses do their work, though, we first need to get a clearer picture of their underlying model.