New-Style Class Changes

You can find formal rationales for this change elsewhere; this book is disinclined to parrot justifications for a change that breaks many working programs. But this is imagined as both an optimization path and a solution to a seemingly obscure call pattern issue. The former rationale is supported by the frequency of built-in operations. If every +, for example, requires extra steps at the instance, it can degrade program speed—especially so given the new-style model’s many attribute-level extensions.

The latter rationale is more obscure, and is described in Python manuals; in short, it reflects a conundrum introduced by the metaclass model. Because classes are now instances of metaclasses, and because metaclasses can define built-in operator methods to process the classes they generate, a method call run for a class must skip the class itself and look one level higher to pick up a method that processes the class, rather than selecting the class’s own version. Its own version would result in an unbound method call, because the class’s own method processes lower instances. This is just the usual unbound method model we discussed in the prior chapter, but is potentially aggravated by the fact that classes can acquire type behavior from metaclasses too.

As a result, because classes are both types and instances in their own right, all instances are skipped for built-in operation method lookup. This is supposedly applied to normal instances for uniformity and consistency, but both non-built-in names and direct and explicit calls to built-in names still check the instance anyhow. Though perhaps a consequence of the new-style class model, to some this may seem a solution arrived at for the sake of a usage pattern that was more artificial and obscure than the widely used one it broke. Its role as optimization path seems more defensible, but also not without repercussions.

In particular, this has potentially broad implications for the delegation-based classes, often known as proxy classes, when embedded objects implement operator overloading. In new-style classes, such a proxy object’s class must generally redefine any such names to catch and delegate, either manually or with tools. The net effect is to either significantly complicate or wholly obviate an entire category of programs. We explored delegation in Chapter 28 and Chapter 31; it’s a common pattern used to augment or adapt another class’s interface—to add validation, tracing, timing, and many other sorts of logic. Though proxies may be more the exception than the rule in typical Python code, many Python programs depend upon them.

In simple terms, and run in Python 2.X to show how new-style classes differ, indexing and prints are routed to __getattr__ in traditional classes, but not for new-style classes, where printing uses a default:^[64]

>>> class C:
        data = 'spam'
        def __getattr__(self, name):             # Classic in 2.X: catches built-ins
            print(name)
            return getattr(self.data, name)

>>> X = C()
>>> X[0]
__getitem__
's'
>>> print(X)                                     # Classic doesn't inherit default
__str__
spam

>>> class C(object):                             # New-style in 2.X and 3.X
        ...rest of class unchanged...

>>> X = C()                                      # Built-ins not routed to getattr
>>> X[0]
TypeError: 'C' object does not support indexing
>>> print(X)
<__main__.C object at 0x02205780>

Though apparently rationalized in the name of class metaclass methods and optimizing built-in operations, this divergence is not addressed by special-casing normal instances having a __getattr__, and applies only to built-in operations—not to normally named methods, or explicit calls to built-in methods by name:

>>> class C: pass                                # 2.X classic class
>>> X = C()
>>> X.normal = lambda: 99
>>> X.normal()
99
>>> X.__add__ = lambda(y): 88 + y
>>> X.__add__(1)
89
>>> X + 1
89

>>> class C(object): pass                        # 2.X/3.X new-style class
>>> X = C()
>>> X.normal = lambda: 99
>>> X.normal()                                   # Normals still from instance
99
>>> X.__add__ = lambda(y): 88 + y
>>> X.__add__(1)                                 # Ditto for explicit built-in names
89
>>> X + 1
TypeError: unsupported operand type(s) for +: 'C' and 'int'

This behavior winds up being inherited by the __getattr__ attribute interception method:

>>> class C(object):
        def __getattr__(self, name): print(name)

>>> X = C()
>>> X.normal             # Normal names are still routed to getattr
normal
>>> X.__add__            # Direct calls by name are too, but expressions are not!
__add__
>>> X + 1
TypeError: unsupported operand type(s) for +: 'C' and 'int'

In a more realistic delegation scenario, this means that built-in operations like expressions no longer work the same as their traditional direct-call equivalent. Asymmetrically, direct calls to built-in method names still work, but equivalent expressions do not because through-type calls fail for names not at the class level and above. In other words, this distinction arises in built-in operations only; explicit fetches run correctly:

>>> class C(object):
        data = 'spam'
        def __getattr__(self, name):
            print('getattr: ' + name)
            return getattr(self.data, name)

>>> X = C()
>>> X.__getitem__(1)           # Traditional mapping works but new-style's does not
getattr: __getitem__
'p'

>>> X[1]
TypeError: 'C' object does not support indexing
>>> type(X).__getitem__(X, 1)
AttributeError: type object 'C' has no attribute '__getitem__'

>>> X.__add__('eggs')          # Ditto for +: instance skipped for expression only
getattr: __add__
'spameggs'

>>> X + 'eggs'
TypeError: unsupported operand type(s) for +: 'C' and 'str'
>>> type(X).__add__(X, 'eggs')
AttributeError: type object 'C' has no attribute '__add__'

The net effect: to code a proxy of an object whose interface may in part be invoked by built-in operations, new-style classes require both __getattr__ for normal names, as well as method redefinitions for all names accessed by built-in operations—whether coded manually, obtained from superclasses, or generated by tools. When redefinitions are so incorporated, calls through both instances and types are equivalent to built-in operations, though redefined names are no longer routed to the generic __getattr__ undefined name handler, even for explicit name calls:

>>> class C(object):                                    # New-style: 3.X and 2.X
        data = 'spam'
        def __getattr__(self, name):                    # Catch normal names
            print('getattr: ' + name)
            return getattr(self.data, name)
        def __getitem__(self, i):                       # Redefine built-ins
            print('getitem: ' + str(i))
            return self.data[i]                         # Run expr or getattr
        def __add__(self, other):
            print('add: ' +  other)
            return getattr(self.data, '__add__')(other)

>>> X = C()
>>> X.upper
getattr: upper
<built-in method upper of str object at 0x0233D670>
>>> X.upper()
getattr: upper
'SPAM'

>>> X[1]                            # Built-in operation (implicit)
getitem: 1
'p'
>>> X.__getitem__(1)                # Traditional equivalence (explicit)
getitem: 1
'p'
>>> type(X).__getitem__(X, 1)       # New-style equivalence
getitem: 1
'p'

>>> X + 'eggs'                      # Ditto for + and others
add: eggs
'spameggs'
>>> X.__add__('eggs')
add: eggs
'spameggs'
>>> type(X).__add__(X, 'eggs')
add: eggs
'spameggs'

We will revisit this change in Chapter 40 on metaclasses, and by example in the contexts of attribute management in Chapter 38 and privacy decorators in Chapter 39. In the latter of these, we’ll also explore coding structures for providing proxies with the required operator methods generically—it’s not an impossible task, and may need to be coded just once if done well. For more of the sort of code influenced by this issue, see those later chapters, as well as the earlier examples in Chapter 28 and Chapter 31.

Because we’ll expand on this issue later in the book, we’ll cut the coverage short here. For external links and pointers on this issue, though, see the following (along with your local search engine):

This story may evolve over time, of course, but has been an issue for many years. As this stands today, classic class proxies for objects that do any operator overloading are effectively broken as new-style classes. Such classes in both 2.X and 3.X require coding or generating wrappers for all the implicitly invoked operator methods a wrapped object may support. This is not ideal for such programs—some proxies may require dozens of wrapper methods (potentially over 50!)—but reflects, or is at least an artifact of, the design goals of new-style class developers.

C:\code> c:\python27\python
>>> class C: pass                       # Classic classes in 2.X

>>> I = C()                             # Instances are made from classes
>>> type(I), I.__class__
(<type 'instance'>, <class __main__.C at 0x02399768>)

>>> type(C)                             # But classes are not the same as types
<type 'classobj'>
>>> C.__class__
AttributeError: class C has no attribute '__class__'

>>> type([1, 2, 3]), [1, 2, 3].__class__
(<type 'list'>, <type 'list'>)

>>> type(list), list.__class__
(<type 'type'>, <type 'type'>)

C:\code> c:\python27\python
>>> class C(object): pass               # New-style classes in 2.X

>>> I = C()                             # Type of instance is class it's made from
>>> type(I), I.__class__
(<class '__main__.C'>, <class '__main__.C'>)

>>> type(C), C.__class__                # Classes are user-defined types
(<type 'type'>, <type 'type'>)

C:\code> c:\python33\python
>>> class C: pass

>>> I = C()                             # All classes are new-style in 3.X
>>> type(I), I.__class__                # Type of instance is class it's made from
(<class '__main__.C'>, <class '__main__.C'>)

>>> type(C), C.__class__                # Class is a type, and type is a class
(<class 'type'>, <class 'type'>)

>>> type([1, 2, 3]), [1, 2, 3].__class__
(<class 'list'>, <class 'list'>)

>>> type(list), list.__class__          # Classes and built-in types work the same
(<class 'type'>, <class 'type'>)

Besides providing for built-in type customization and metaclass hooks, the merging of classes and types in the new-style class model can impact code that does type testing. In Python 3.X, for example, the types of class instances compare directly and meaningfully, and in the same way as built-in type objects. This follows from the fact that classes are now types, and an instance’s type is the instance’s class:

C:\code> c:\python33\python
>>> class C: pass
>>> class D: pass

>>> c, d = C(), D()
>>> type(c) == type(d)                 # 3.X: compares the instances' classes
False

>>> type(c), type(d)
(<class '__main__.C'>, <class '__main__.D'>)
>>> c.__class__, d.__class__
(<class '__main__.C'>, <class '__main__.D'>)

>>> c1, c2 = C(), C()
>>> type(c1) == type(c2)
True

With classic classes in 2.X, though, comparing instance types is almost useless, because all instances have the same “instance” type. To truly compare types, the instance __class__ attributes must be compared (if you care about portability, this works in 3.X, too, but it’s not required there):

C:\code> c:\python27\python
>>> class C: pass
>>> class D: pass

>>> c, d = C(), D()
>>> type(c) == type(d)                 # 2.X: all instances are same type!
True
>>> c.__class__ == d.__class__         # Compare classes explicitly if needed
False

>>> type(c), type(d)
(<type 'instance'>, <type 'instance'>)
>>> c.__class__, d.__class__
(<class __main__.C at 0x024585A0>, <class __main__.D at 0x024588D0>)

And as you should expect by now, new-style classes in 2.X work the same as all classes in 3.X in this regard—comparing instance types compares the instances’ classes automatically:

C:\code> c:\python27\python
>>> class C(object): pass
>>> class D(object): pass

>>> c, d = C(), D()
>>> type(c) == type(d)                 # 2.X new-style: same as all in 3.X
False

>>> type(c), type(d)
(<class '__main__.C'>, <class '__main__.D'>)
>>> c.__class__, d.__class__
(<class '__main__.C'>, <class '__main__.D'>)

Of course, as I’ve pointed out numerous times in this book, type checking is usually the wrong thing to do in Python programs (we code to object interfaces, not object types), and the more general isinstance built-in is more likely what you’ll want to use in the rare cases where instance class types must be queried. However, knowledge of Python’s type model can help clarify the class model in general.

>>> class C: pass                     # For new-style classes
>>> X = C()
>>> type(X), type(C)                  # Type is class instance was created from
(<class '__main__.C'>, <class 'type'>)

>>> isinstance(X, object)
True
>>> isinstance(C, object)             # Classes always inherit from object
True

>>> type('spam'), type(str)
(<class 'str'>, <class 'type'>)

>>> isinstance('spam', object)        # Same for  built-in types (classes)
True
>>> isinstance(str, object)
True

>>> type(type)                        # All classes are types, and vice versa
<class 'type'>
>>> type(object)
<class 'type'>

>>> isinstance(type, object)          # All classes derive from object, even type
True
>>> isinstance(object, type)          # Types make classes, and type is a class
True
>>> type is object
False

The preceding may seem obscure, but this model has a number of practical implications. For one thing, it means that we sometimes must be aware of the method defaults that come with the explicit or implicit object root class in new-style classes only:

c:\code> py −2
>>> dir(object)
['__class__', '__delattr__', '__doc__', '__format__', '__getattribute__', '__hash__'
, '__init__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '
__sizeof__', '__str__', '__subclasshook__']

>>> class C: pass
>>> C.__bases__                       # Classic classes do not inherit from object
()
>>> X = C()
>>> X.__repr__
AttributeError: C instance has no attribute '__repr__'

>>> class C(object): pass             # New-style classes inherit object defaults
>>> C.__bases__
(<type 'object'>,)
>>> X = C()
>>> X.__repr__
<method-wrapper '__repr__' of C object at 0x00000000020B5978>

c:\code> py −3
>>> class C: pass                     # This means all classes get defaults in 3.X
>>> C.__bases__
(<class 'object'>,)
>>> C().__repr__
<method-wrapper '__repr__' of C object at 0x0000000002955630>

This model also makes for fewer special cases than the prior type/class distinction of classic classes, and it allows us to write code that can safely assume and use an object superclass (e.g., by assuming it as an “anchor” in some super built-in roles described ahead, and by passing it method calls to invoke default behavior). We’ll see examples of the latter later in the book; for now, let’s move on to explore the last major new-style change.

To illustrate how the new-style MRO search differs, consider this simplistic incarnation of the diamond multiple inheritance pattern for classic classes. Here, D’s superclasses B and C both lead to the same common ancestor, A:

>>> class A:       attr = 1           # Classic (Python 2.X)
>>> class B(A):    pass               # B and C both lead to A
>>> class C(A):    attr = 2
>>> class D(B, C): pass               # Tries A before C

>>> x = D()
>>> x.attr                            # Searches x, D, B, A
1

The attribute x.attr here is found in superclass A, because with classic classes, the inheritance search climbs as high as it can before backing up and moving right. The full DFLR search order would visit x, D, B, A, C, and then A. For this attribute, the search stops as soon as attr is found in A, above B.

However, with new-style classes derived from a built-in like object (and all classes in 3.X), the search order is different: Python looks in C to the right of B, before trying A above B. The full MRO search order would visit x, D, B, C, and then A. For this attribute, the search stops as soon as attr is found in C:

>>> class A(object): attr = 1         # New-style ("object" not required in 3.X)
>>> class B(A):      pass
>>> class C(A):      attr = 2
>>> class D(B, C):   pass             # Tries C before A

>>> x = D()
>>> x.attr                            # Searches x, D, B, C
2

This change in the inheritance search procedure is based upon the assumption that if you mix in C lower in the tree, you probably intend to grab its attributes in preference to A’s. It also assumes that C is always intended to override A’s attributes in all contexts, which is probably true when it’s used standalone but may not be when it’s mixed into a diamond with classic classes—you might not even know that C may be mixed in like this when you code it.

Since it is most likely that the programmer meant that C should override A in this case, though, new-style classes visit C first. Otherwise, C could be essentially pointless in a diamond context for any names in A too—it could not customize A and would be used only for names unique to C.

Of course, the problem with assumptions is that they assume things! If this search order deviation seems too subtle to remember, or if you want more control over the search process, you can always force the selection of an attribute from anywhere in the tree by assigning or otherwise naming the one you want at the place where the classes are mixed together. The following, for example, chooses new-style order in a classic class by resolving the choice explicitly:

>>> class A:       attr = 1           # Classic
>>> class B(A):    pass
>>> class C(A):    attr = 2
>>> class D(B, C): attr = C.attr      # <== Choose C, to the right

>>> x = D()
>>> x.attr                            # Works like new-style (all 3.X)
2

Here, a tree of classic classes is emulating the search order of new-style classes for a specific attribute: the assignment to the attribute in D picks the version in C, thereby subverting the normal inheritance search path (D.attr will be lowest in the tree). New-style classes can similarly emulate classic classes by choosing the higher version of the target attribute at the place where the classes are mixed together:

>>> class A(object): attr = 1         # New-style
>>> class B(A):      pass
>>> class C(A):      attr = 2
>>> class D(B, C):   attr = B.attr    # <== Choose A.attr, above

>>> x = D()
>>> x.attr                            # Works like classic (default 2.X)
1

If you are willing to always resolve conflicts like this, you may be able to largely ignore the search order difference and not rely on assumptions about what you meant when you coded your classes.

Naturally, attributes picked this way can also be method functions—methods are normal, assignable attributes that happen to reference callable function objects:

>>> class A:
        def meth(s): print('A.meth')

>>> class C(A):
        def meth(s): print('C.meth')

>>> class B(A):
        pass

>>> class D(B, C): pass               # Use default search order
>>> x = D()                           # Will vary per class type
>>> x.meth()                          # Defaults to classic order in 2.X
A.meth

>>> class D(B, C): meth = C.meth      # <== Pick C's method: new-style (and 3.X)
>>> x = D()
>>> x.meth()
C.meth

>>> class D(B, C): meth = B.meth      # <== Pick B's method: classic
>>> x = D()
>>> x.meth()
A.meth

Here, we select methods by explicitly assigning to names lower in the tree. We might also simply call the desired class explicitly; in practice, this pattern might be more common, especially for things like constructors:

class D(B, C):
    def meth(self):                   # Redefine lower
        ...
        C.meth(self)                  # <== Pick C's method by calling

Such selections by assignment or call at mix-in points can effectively insulate your code from this difference in class flavors. This applies only to the attributes you handle this way, of course, but explicitly resolving the conflicts ensures that your code won’t vary per Python version, at least in terms of attribute conflict selection. In other words, this can serve as a portability technique for classes that may need to be run under both the new-style and classic class models.

In sum, by default, the diamond pattern is searched differently for classic and new-style classes, and this is a non-backward-compatible change. Keep in mind, though, that this change primarily affects diamond pattern cases of multiple inheritance; new-style class inheritance works the same for most other inheritance tree structures. Further, it’s not impossible that this entire issue may be of more theoretical than practical importance—because the new-style search wasn’t significant enough to address until Python 2.2 and didn’t become standard until 3.0, it seems unlikely to impact most Python code.

Having said that, I should also note that even though you might not code diamond patterns in classes you write yourself, because the implied object superclass is above every root class in 3.X as we saw earlier, every case of multiple inheritance exhibits the diamond pattern today. That is, in new-style classes, object automatically plays the role that the class A does in the example we just considered. Hence the new-style MRO search rule not only modifies logical semantics, but is also an important performance optimization—it avoids visiting and searching the same class more than once, even the automatic object.

Just as important, we’ve also seen that the implied object superclass in the new-style model provides default methods for a variety of built-in operations, including the __str__ and __repr__ display format methods. Run a dir(object) to see which methods are provided. Without the new-style MRO search order, in multiple inheritance cases the defaults in object would always override redefinitions in user-coded classes, unless they were always made in the leftmost superclass. In other words, the new-style class model itself makes using the new-style search order more critical!

For a more visual example of the implied object superclass in 3.X, and other examples of diamond patterns created by it, see the ListTree class’s output in the lister.py example in the preceding chapter, as well as the classtree.py tree walker example in Chapter 29—and the next section.

This book avoids a full description of the MRO algorithm deliberately, because many Python programmers don’t need to care (this only impacts diamonds, which are relatively rare in real-world code); because it differs between 2.X and 3.X; and because the details of the MRO are a bit too arcane and academic for this text. As a rule, this book avoids formal algorithms and prefers to teach informally by example.

On the other hand, some readers may still have an interest in the formal theory behind new-style MRO. If this set includes you, it’s described in full detail online; search Python’s manuals and the Web for current MRO links. In short, though, the MRO essentially works like this:

The resulting MRO list for a given class includes the class, its superclasses, and all higher superclasses up to the object root class at the top of the tree. It’s ordered such that each class appears before its parents, and multiple parents retain the order in which they appear in the __bases__ superclass tuple.

Crucially, though, because common parents in diamonds appear only at the position of their last visitation, lower classes are searched first when the MRO list is later used by attribute inheritance. Moreover, each class is included and thus visited just once, no matter how many classes lead to it.

We’ll see applications of this algorithm later in this chapter, including that in super—a built-in that elevates the MRO to required reading if you wish to fully understand how methods are dispatched by this call, should you choose to use it. As we’ll see, despite its name, this call invokes the next class on the MRO, which might not be a superclass at all.

If you just want to see how Python’s new-style inheritance orders superclasses in general, though, new-style classes (and hence all classes in 3.X) have a class.__mro__ attribute, which is a tuple giving the linear search order Python uses to look up attributes in superclasses. Really, this attribute is the inheritance order in new-style classes, and is often as much MRO detail as many Python users need.

Here are some illustrative examples, run in 3.X; for diamond inheritance patterns only, the search is the new order we’ve been studying—across before up, per the MRO for new-style classes always used in 3.X, and available as an option in 2.X:

>>> class A: pass
>>> class B(A): pass         # Diamonds: order differs for newstyle
>>> class C(A): pass         # Breadth-first across lower levels
>>> class D(B, C): pass
>>> D.__mro__
(<class '__main__.D'>, <class '__main__.B'>, <class '__main__.C'>,
<class '__main__.A'>, <class 'object'>)

For nondiamonds, though, the search is still as it has always been (albeit with an extra object root)—to the top, and then to the right (a.k.a. DFLR, depth first and left to right, the model used for all classic classes in 2.X):

>>> class A: pass
>>> class B(A): pass         # Nondiamonds: order same as classic
>>> class C: pass            # Depth first, then left to right
>>> class D(B, C): pass
>>> D.__mro__
(<class '__main__.D'>, <class '__main__.B'>, <class '__main__.A'>,
<class '__main__.C'>, <class 'object'>)

The MRO of the following tree, for example, is the same as the earlier diamond, per DFLR:

>>> class A: pass
>>> class B: pass            # Another nondiamond: DFLR
>>> class C(A): pass
>>> class D(B, C): pass
>>> D.__mro__
(<class '__main__.D'>, <class '__main__.B'>, <class '__main__.C'>,
<class '__main__.A'>, <class 'object'>)

Notice how the implied object superclass always shows up at the end of the MRO; as we’ve seen, it’s added automatically above root (topmost) classes in new-style class trees in 3.X (and optionally in 2.X):

>>> A.__bases__              # Superclass links: object at two roots
(<class 'object'>,)
>>> B.__bases__
(<class 'object'>,)
>>> C.__bases__
(<class '__main__.A'>,)
>>> D.__bases__
(<class '__main__.B'>, <class '__main__.C'>)

Technically, the implied object superclass always creates a diamond in multiple inheritance even if your classes do not—your classes are searched as before, but the new-style MRO ensures that object is visited last, so your classes can override its defaults:

>>> class X: pass
>>> class Y: pass
>>> class A(X): pass         # Nondiamond: depth first then left to right
>>> class B(Y): pass         # Though implied "object" always forms a diamond
>>> class D(A, B): pass
>>> D.mro()
[<class '__main__.D'>, <class '__main__.A'>, <class '__main__.X'>,
<class '__main__.B'>, <class '__main__.Y'>, <class 'object'>]

>>> X.__bases__, Y.__bases__
((<class 'object'>,), (<class 'object'>,))
>>> A.__bases__, B.__bases__
((<class '__main__.X'>,), (<class '__main__.Y'>,))

The class.__mro__ attribute is available only on new-style classes; it’s not present in 2.X unless classes derive from object. Strictly speaking, new-style classes also have a class.mro() method used in the prior example for variety; it’s called at class instantiation time and its return value is a list used to initialize the __mro__ attribute when the class is created (the method is available for customization in metaclasses, described later). You can also select MRO names if classes’ object displays are too detailed, though this book usually shows the objects to remind you of their true form:

>>> D.mro() == list(D.__mro__)
True
>>> [cls.__name__ for cls in D.__mro__]
['D', 'A', 'X', 'B', 'Y', 'object']

However you access or display them, class MRO paths might be useful to resolve confusion, and in tools that must imitate Python’s inheritance search order. The next section shows the latter role in action.