Step 6: Using Introspection Tools

We can address both issues with Python’s introspection tools—special attributes and functions that give us access to some of the internals of objects’ implementations. These tools are somewhat advanced and generally used more by people writing tools for other programmers to use than by programmers developing applications. Even so, a basic knowledge of some of these tools is useful because they allow us to write code that processes classes in generic ways. In our code, for example, there are two hooks that can help us out, both of which were introduced near the end of the preceding chapter and used in earlier examples:

We met the first of these categories in the prior chapter, but here’s a quick review at Python’s interactive prompt with the latest versions of our person.py classes. Notice how we load Person at the interactive prompt with a from statement here—class names live in and are imported from modules, exactly like function names and other variables:

>>> from person import Person
>>> bob = Person('Bob Smith')
>>> bob                                        # Show bob's __repr__ (not __str__)
[Person: Bob Smith, 0]
>>> print(bob)                                 # Ditto: print => __str__ or __repr__
[Person: Bob Smith, 0]

>>> bob.__class__                              # Show bob's class and its name
<class 'person.Person'>
>>> bob.__class__.__name__
'Person'

>>> list(bob.__dict__.keys())                  # Attributes are really dict keys
['pay', 'job', 'name']                         # Use list to force list in 3.X

>>> for key in bob.__dict__:
        print(key, '=>', bob.__dict__[key])    # Index manually

pay => 0
job => None
name => Bob Smith

>>> for key in bob.__dict__:
        print(key, '=>', getattr(bob, key))    # obj.attr, but attr is a var

pay => 0
job => None
name => Bob Smith

As noted briefly in the prior chapter, some attributes accessible from an instance might not be stored in the __dict__ dictionary if the instance’s class defines __slots__: an optional and relatively obscure feature of new-style classes (and hence all classes in Python 3.X) that stores attributes sequentially in the instance; may preclude an instance __dict__ altogether; and which we won’t study in full until Chapter 31 and Chapter 32. Since slots really belong to classes instead of instances, and since they are rarely used in any event, we can reasonably ignore them here and focus on the normal __dict__.

As we do, though, keep in mind that some programs may need to catch exceptions for a missing __dict__, or use hasattr to test or getattr with a default if its users might deploy slots. As we’ll see in Chapter 32, the next section’s code won’t fail if used by a class with slots (its lack of them is enough to guarantee a __dict__) but slots—and other “virtual” attributes—won’t be reported as instance data.

We can put these interfaces to work in a superclass that displays accurate class names and formats all attributes of an instance of any class. Open a new file in your text editor to code the following—it’s a new, independent module named classtools.py that implements just such a class. Because its __repr__ display overload uses generic introspection tools, it will work on any instance, regardless of the instance’s attributes set. And because this is a class, it automatically becomes a general formatting tool: thanks to inheritance, it can be mixed into any class that wishes to use its display format. As an added bonus, if we ever want to change how instances are displayed we need only change this class, as every class that inherits its __repr__ will automatically pick up the new format when it’s next run:

# File classtools.py (new)
"Assorted class utilities and tools"

class AttrDisplay:
    """
    Provides an inheritable display overload method that shows
    instances with their class names and a name=value pair for
    each attribute stored on the instance itself (but not attrs
    inherited from its classes). Can be mixed into any class,
    and will work on any instance.
    """
    def gatherAttrs(self):
        attrs = []
        for key in sorted(self.__dict__):
            attrs.append('%s=%s' % (key, getattr(self, key)))
        return ', '.join(attrs)

    def __repr__(self):
        return '[%s: %s]' % (self.__class__.__name__, self.gatherAttrs())

if __name__ == '__main__':

    class TopTest(AttrDisplay):
        count = 0
        def __init__(self):
            self.attr1 = TopTest.count
            self.attr2 = TopTest.count+1
            TopTest.count += 2

    class SubTest(TopTest):
        pass

    X, Y = TopTest(), SubTest()      # Make two instances
    print(X)                         # Show all instance attrs
    print(Y)                         # Show lowest class name

Notice the docstrings here—because this is a general-purpose tool, we want to add some functional documentation for potential users to read. As we saw in Chapter 15, docstrings can be placed at the top of simple functions and modules, and also at the start of classes and any of their methods; the help function and the PyDoc tool extract and display these automatically. We’ll revisit docstrings for classes in Chapter 29.

When run directly, this module’s self-test makes two instances and prints them; the __repr__ defined here shows the instance’s class, and all its attributes names and values, in sorted attribute name order. This output is the same in Python 3.X and 2.X because each object’s display is a single constructed string:

C:\code> classtools.py
[TopTest: attr1=0, attr2=1]
[SubTest: attr1=2, attr2=3]

Another design note here: because this class uses __repr__ instead of __str__ its displays are used in all contexts, but its clients also won’t have the option of providing an alternative low-level display—they can still add a __str__, but this applies to print and str only. In a more general tool, using __str__ instead limits a display’s scope, but leaves clients the option of adding a __repr__ for a secondary display at interactive prompts and nested appearances. We’ll follow this alternative policy when we code expanded versions of this class in Chapter 31; for this demo, we’ll stick with the all-inclusive __repr__.

If you study the classtools module’s self-test code long enough, you’ll notice that its class displays only instance attributes, attached to the self object at the bottom of the inheritance tree; that’s what self’s __dict__ contains. As an intended consequence, we don’t see attributes inherited by the instance from classes above it in the tree (e.g., count in this file’s self-test code—a class attribute used as an instance counter). Inherited class attributes are attached to the class only, not copied down to instances.

If you ever do wish to include inherited attributes too, you can climb the __class__ link to the instance’s class, use the __dict__ there to fetch class attributes, and then iterate through the class’s __bases__ attribute to climb to even higher superclasses, repeating as necessary. If you’re a fan of simple code, running a built-in dir call on the instance instead of using __dict__ and climbing would have much the same effect, since dir results include inherited names in the sorted results list. In Python 2.7:

>>> from person import Person           # 2.X: keys is list, dir shows less
>>> bob = Person('Bob Smith')

>>> bob.__dict__.keys()                 # Instance attrs only
['pay', 'job', 'name']

>>> dir(bob)                            # Plus inherited attrs in classes
['__doc__', '__init__', '__module__', '__repr__', 'giveRaise', 'job', 'lastName',
'name', 'pay']

If you’re using Python 3.X, your output will vary, and may be more than you bargained for; here’s the 3.3 result for the last two statements (keys list order can vary per run):

>>> list(bob.__dict__.keys())           # 3.X keys is a view, not a list
['name', 'job', 'pay']

>>> dir(bob)                            # 3.X includes class type methods
['__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__',
'__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__',
...more omitted: 31 attrs...
'__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__',
'giveRaise', 'job', 'lastName', 'name', 'pay']

The code and output here varies between Python 2.X and 3.X, because 3.X’s dict.keys is not a list, and 3.X’s dir returns extra class-type implementation attributes. Technically, dir returns more in 3.X because classes are all “new style” and inherit a large set of operator overloading names from the class type. In fact, as usual you’ll probably want to filter out most of the __X__ names in the 3.X dir result, since they are internal implementation details and not something you’d normally want to display:

>>> len(dir(bob))
31
>>> list(name for name in dir(bob) if not name.startswith('__'))
['giveRaise', 'job', 'lastName', 'name', 'pay']

In the interest of space, we’ll leave optional display of inherited class attributes with either tree climbs or dir as suggested experiments for now. For more hints on this front, though, watch for the classtree.py inheritance tree climber we will write in Chapter 29, and the lister.py attribute listers and climbers we’ll code in Chapter 31.

One last subtlety here: because our AttrDisplay class in the classtools module is a general tool designed to be mixed into other arbitrary classes, we have to be aware of the potential for unintended name collisions with client classes. As is, I’ve assumed that client subclasses may want to use both its __repr__ and gatherAttrs, but the latter of these may be more than a subclass expects—if a subclass innocently defines a gatherAttrs name of its own, it will likely break our class, because the lower version in the subclass will be used instead of ours.

To see this for yourself, add a gatherAttrs to TopTest in the file’s self-test code; unless the new method is identical, or intentionally customizes the original, our tool class will no longer work as planned—self.gatherAttrs within AttrDisplay searches anew from the TopTest instance:

    class TopTest(AttrDisplay):
        ....
        def gatherAttrs(self):         # Replaces method in AttrDisplay!
            return 'Spam'

This isn’t necessarily bad—sometimes we want other methods to be available to subclasses, either for direct calls or for customization this way. If we really meant to provide a __repr__ only, though, this is less than ideal.

To minimize the chances of name collisions like this, Python programmers often prefix methods not meant for external use with a single underscore: _gatherAttrs in our case. This isn’t foolproof (what if another class defines _gatherAttrs, too?), but it’s usually sufficient, and it’s a common Python naming convention for methods internal to a class.

A better and less commonly used solution would be to use two underscores at the front of the method name only: __gatherAttrs for us. Python automatically expands such names to include the enclosing class’s name, which makes them truly unique when looked up by the inheritance search. This is a feature usually called pseudoprivate class attributes, which we’ll expand on in Chapter 31 and deploy in an expanded version of this class there. For now, we’ll make both our methods available.

Now, to use this generic tool in our classes, all we need to do is import it from its module, mix it in by inheritance in our top-level class, and get rid of the more specific __repr__ we coded before. The new display overload method will be inherited by instances of Person, as well as Manager; Manager gets __repr__ from Person, which now obtains it from the AttrDisplay coded in another module. Here is the final version of our person.py file with these changes applied:

# File classtools.py (new)
...as listed earlier...

# File person.py (final)
"""
Record and process information about people.
Run this file directly to test its classes.
"""
from classtools import AttrDisplay                    # Use generic display tool

class Person(AttrDisplay):                            # Mix in a repr at this level
    """
    Create and process person records
    """
    def __init__(self, name, job=None, pay=0):
        self.name = name
        self.job  = job
        self.pay  = pay

    def lastName(self):                               # Assumes last is last
        return self.name.split()[-1]

    def giveRaise(self, percent):                     # Percent must be 0..1
        self.pay = int(self.pay * (1 + percent))

class Manager(Person):
    """
    A customized Person with special requirements
    """
    def __init__(self, name, pay):
        Person.__init__(self, name, 'mgr', pay)       # Job name is implied

    def giveRaise(self, percent, bonus=.10):
        Person.giveRaise(self, percent + bonus)

if __name__ == '__main__':
    bob = Person('Bob Smith')
    sue = Person('Sue Jones', job='dev', pay=100000)
    print(bob)
    print(sue)
    print(bob.lastName(), sue.lastName())
    sue.giveRaise(.10)
    print(sue)
    tom = Manager('Tom Jones', 50000)
    tom.giveRaise(.10)
    print(tom.lastName())
    print(tom)

As this is the final revision, we’ve added a few comments here to document our work—docstrings for functional descriptions and # for smaller notes, per best-practice conventions, as well as blank lines between methods for readability—a generally good style choice when classes or methods grow large, which I resisted earlier for these small classes, in part to save space and keep the code more compact.

When we run this code now, we see all the attributes of our objects, not just the ones we hardcoded in the original __repr__. And our final issue is resolved: because AttrDisplay takes class names off the self instance directly, each object is shown with the name of its closest (lowest) class—tom displays as a Manager now, not a Person, and we can finally verify that his job name has been correctly filled in by the Manager constructor:

C:\code> person.py
[Person: job=None, name=Bob Smith, pay=0]
[Person: job=dev, name=Sue Jones, pay=100000]
Smith Jones
[Person: job=dev, name=Sue Jones, pay=110000]
Jones
[Manager: job=mgr, name=Tom Jones, pay=60000]

This is the more useful display we were after. From a larger perspective, though, our attribute display class has become a general tool, which we can mix into any class by inheritance to leverage the display format it defines. Further, all its clients will automatically pick up future changes in our tool. Later in the book, we’ll meet even more powerful class tool concepts, such as decorators and metaclasses; along with Python’s many introspection tools, they allow us to write code that augments and manages classes in structured and maintainable ways.