Namespaces: The Conclusion

As we’ve learned, unqualified simple names follow the LEGB lexical scoping rule outlined when we explored functions in Chapter 17:

Also per Chapter 17, some special-case constructs localize names further (e.g., variables in some comprehensions and try statement clauses), but the vast majority of names follow the LEGB rule.

We’ve also seen that qualified attribute names refer to attributes of specific objects and obey the rules for modules and classes. For class and instance objects, the reference rules are augmented to include the inheritance search procedure:

As noted earlier, the preceding captures the normal and typical case. These attribute rules can vary in classes that utilize more advanced tools, especially for new-style classes—an option in 2.X and the standard in 3.X, which we’ll explore in Chapter 32. For example, reference inheritance can be richer than implied here when metaclasses are deployed, and classes which leverage attribute management tools such as properties, descriptors, and __setattr__ can intercept and route attribute assignments arbitrarily.

In fact, some inheritance is run on assignment too, to locate descriptors with a __set__ method in new-style classes; such tools override the normal rules for both reference and assignment. We’ll explore attribute management tools in depth in Chapter 38, and formalize inheritance and its use of descriptors in Chapter 40. For now, most readers should focus on the normal rules given here, which cover most Python application code.

With distinct search procedures for qualified and unqualified names, and multiple lookup layers for both, it can sometimes be difficult to tell where a name will wind up going. In Python, the place where you assign a name is crucial—it fully determines the scope or object in which a name will reside. The file manynames.py illustrates how this principle translates to code and summarizes the namespace ideas we have seen throughout this book (sans obscure special-case scopes like comprehensions):

# File manynames.py

X = 11                       # Global (module) name/attribute (X, or manynames.X)

def f():
    print(X)                 # Access global X (11)

def g():
    X = 22                   # Local (function) variable (X, hides module X)
    print(X)

class C:
    X = 33                   # Class attribute (C.X)
    def m(self):
        X = 44               # Local variable in method (X)
        self.X = 55          # Instance attribute (instance.X)

This file assigns the same name, X, five times—illustrative, though not exactly best practice! Because this name is assigned in five different locations, though, all five Xs in this program are completely different variables. From top to bottom, the assignments to X here generate: a module attribute (11), a local variable in a function (22), a class attribute (33), a local variable in a method (44), and an instance attribute (55). Although all five are named X, the fact that they are all assigned at different places in the source code or to different objects makes all of these unique variables.

You should take the time to study this example carefully because it collects ideas we’ve been exploring throughout the last few parts of this book. When it makes sense to you, you will have achieved Python namespace enlightenment. Or, you can run the code and see what happens—here’s the remainder of this source file, which makes an instance and prints all the Xs that it can fetch:

# manynames.py, continued

if __name__ == '__main__':
    print(X)                 # 11: module (a.k.a. manynames.X outside file)
    f()                      # 11: global
    g()                      # 22: local
    print(X)                 # 11: module name unchanged

    obj = C()                # Make instance
    print(obj.X)             # 33: class name inherited by instance

    obj.m()                  # Attach attribute name X to instance now
    print(obj.X)             # 55: instance
    print(C.X)               # 33: class (a.k.a. obj.X if no X in instance)

    #print(C.m.X)            # FAILS: only visible in method
    #print(g.X)              # FAILS: only visible in function

The outputs that are printed when the file is run are noted in the comments in the code; trace through them to see which variable named X is being accessed each time. Notice in particular that we can go through the class to fetch its attribute (C.X), but we can never fetch local variables in functions or methods from outside their def statements. Locals are visible only to other code within the def, and in fact only live in memory while a call to the function or method is executing.

Some of the names defined by this file are visible outside the file to other modules too, but recall that we must always import before we can access names in another file—name segregation is the main point of modules, after all:

# otherfile.py

import manynames

X = 66
print(X)                     # 66: the global here
print(manynames.X)           # 11: globals become attributes after imports

manynames.f()                # 11: manynames's X, not the one here!
manynames.g()                # 22: local in other file's function

print(manynames.C.X)         # 33: attribute of class in other module
I = manynames.C()
print(I.X)                   # 33: still from class here
I.m()
print(I.X)                   # 55: now from instance!

Notice here how manynames.f() prints the X in manynames, not the X assigned in this file—scopes are always determined by the position of assignments in your source code (i.e., lexically) and are never influenced by what imports what or who imports whom. Also, notice that the instance’s own X is not created until we call I.m()—attributes, like all variables, spring into existence when assigned, and not before. Normally we create instance attributes by assigning them in class __init__ constructor methods, but this isn’t the only option.

Finally, as we learned in Chapter 17, it’s also possible for a function to change names outside itself, with global and (in Python 3.X) nonlocal statements—these statements provide write access, but also modify assignment’s namespace binding rules:

X = 11                       # Global in module

def g1():
    print(X)                 # Reference global in module (11)

def g2():
    global X
    X = 22                   # Change global in module

def h1():
    X = 33                   # Local in function
    def nested():
        print(X)             # Reference local in enclosing scope (33)

def h2():
    X = 33                   # Local in function
    def nested():
        nonlocal X           # Python 3.X statement
        X = 44               # Change local in enclosing scope

Of course, you generally shouldn’t use the same name for every variable in your script—but as this example demonstrates, even if you do, Python’s namespaces will work to keep names used in one context from accidentally clashing with those used in another.

The preceding example summarized the effect of nested functions on scopes, which we studied in Chapter 17. It turns out that classes can be nested too—a useful coding pattern in some types of programs, with scope implications that follow naturally from what you already know, but that may not be obvious on first encounter. This section illustrates the concept by example.

Though they are normally coded at the top level of a module, classes also sometimes appear nested in functions that generate them—a variation on the “factory function” (a.k.a. closure) theme in Chapter 17, with similar state retention roles. There we noted that class statements introduce new local scopes much like function def statements, which follow the same LEGB scope lookup rule as function definitions.

This rule applies both to the top level of the class itself, as well as to the top level of method functions nested within it. Both form the L layer in this rule—they are normal local scopes, with access to their names, names in any enclosing functions, globals in the enclosing module, and built-ins. Like modules, the class’s local scope morphs into an attribute namespace after the class statement is run.

Although classes have access to enclosing functions’ scopes, though, they do not act as enclosing scopes to code nested within the class: Python searches enclosing functions for referenced names, but never any enclosing classes. That is, a class is a local scope and has access to enclosing local scopes, but it does not serve as an enclosing local scope to further nested code. Because the search for names used in method functions skips the enclosing class, class attributes must be fetched as object attributes using inheritance.

For example, in the following nester function, all references to X are routed to the global scope except the last, which picks up a local scope redefinition (the section’s code is in file classscope.py, and the output of each example is described in its last two comments):

X = 1

def nester():
   print(X)                 # Global: 1
   class C:
       print(X)             # Global: 1
       def method1(self):
           print(X)         # Global: 1
       def method2(self):
           X = 3            # Hides global
           print(X)         # Local: 3
   I = C()
   I.method1()
   I.method2()

print(X)                    # Global: 1
nester()                    # Rest: 1, 1, 1, 3
print('-'*40)

Watch what happens, though, when we reassign the same name in nested function layers: the redefinitions of X create locals that hide those in enclosing scopes, just as for simple nested functions; the enclosing class layer does not change this rule, and in fact is irrelevant to it:

X = 1

def nester():
   X = 2                    # Hides global
   print(X)                 # Local: 2
   class C:
       print(X)             # In enclosing def (nester): 2
       def method1(self):
           print(X)         # In enclosing def (nester): 2
       def method2(self):
           X = 3            # Hides enclosing (nester)
           print(X)         # Local: 3
   I = C()
   I.method1()
   I.method2()

print(X)                    # Global: 1
nester()                    # Rest: 2, 2, 2, 3
print('-'*40)

And here’s what happens when we reassign the same name at multiple stops along the way: assignments in the local scopes of both functions and classes hide globals or enclosing function locals of the same name, regardless of the nesting involved:

X = 1

def nester():
   X = 2                    # Hides global
   print(X)                 # Local: 2
   class C:
       X = 3                # Class local hides nester's: C.X or I.X (not scoped)
       print(X)             # Local: 3
       def method1(self):
           print(X)         # In enclosing def (not 3 in class!): 2
           print(self.X)    # Inherited class local: 3
       def method2(self):
           X = 4            # Hides enclosing (nester, not class)
           print(X)         # Local: 4
           self.X = 5       # Hides class
           print(self.X)    # Located in instance: 5
   I = C()
   I.method1()
   I.method2()

print(X)                    # Global: 1
nester()                    # Rest: 2, 3, 2, 3, 4, 5
print('-'*40)

Most importantly, the lookup rules for simple names like X never search enclosing class statements—just defs, modules, and built-ins (it’s the LEGB rule, not CLEGB!). In method1, for example, X is found in a def outside the enclosing class that has the same name in its local scope. To get to names assigned in the class (e.g., methods), we must fetch them as class or instance object attributes, via self.X in this case.

Believe it or not, we’ll see use cases for this nested classes coding pattern later in this book, especially in some of Chapter 39’s decorators. In this role, the enclosing function usually both serves as a class factory and provides retained state for later use in the enclosed class or its methods.

In Chapter 23, we learned that module namespaces have a concrete implementation as dictionaries, exposed with the built-in __dict__ attribute. In Chapter 27 and Chapter 28, we learned that the same holds true for class and instance objects—attribute qualification is mostly a dictionary indexing operation internally, and attribute inheritance is largely a matter of searching linked dictionaries. In fact, within Python, instance and class objects are mostly just dictionaries with links between them. Python exposes these dictionaries, as well as their links, for use in advanced roles (e.g., for coding tools).

We put some of these tools to work in the prior chapter, but to summarize and help you better understand how attributes work internally, let’s work through an interactive session that traces the way namespace dictionaries grow when classes are involved. Now that we know more about methods and superclasses, we can also embellish the coverage here for a better look. First, let’s define a superclass and a subclass with methods that will store data in their instances:

>>> class Super:
        def hello(self):
            self.data1 = 'spam'

>>> class Sub(Super):
        def hola(self):
            self.data2 = 'eggs'

When we make an instance of the subclass, the instance starts out with an empty namespace dictionary, but it has links back to the class for the inheritance search to follow. In fact, the inheritance tree is explicitly available in special attributes, which you can inspect. Instances have a __class__ attribute that links to their class, and classes have a __bases__ attribute that is a tuple containing links to higher superclasses (I’m running this on Python 3.3; your name formats, internal attributes, and key orders may vary):

>>> X = Sub()
>>> X.__dict__                            # Instance namespace dict
{}
>>> X.__class__                           # Class of instance
<class '__main__.Sub'>
>>> Sub.__bases__                         # Superclasses of class
(<class '__main__.Super'>,)
>>> Super.__bases__                       # () empty tuple in Python 2.X
(<class 'object'>,)

As classes assign to self attributes, they populate the instance objects—that is, attributes wind up in the instances’ attribute namespace dictionaries, not in the classes’. An instance object’s namespace records data that can vary from instance to instance, and self is a hook into that namespace:

>>> Y = Sub()

>>> X.hello()
>>> X.__dict__
{'data1': 'spam'}

>>> X.hola()
>>> X.__dict__
{'data2': 'eggs', 'data1': 'spam'}

>>> list(Sub.__dict__.keys())
['__qualname__', '__module__', '__doc__', 'hola']
>>> list(Super.__dict__.keys())
['__module__', 'hello', '__dict__', '__qualname__', '__doc__', '__weakref__']

>>> Y.__dict__
{}

Notice the extra underscore names in the class dictionaries; Python sets these automatically, and we can filter them out with the generator expressions we saw in Chapter 27 and Chapter 28 that we won’t repeat here. Most are not used in typical programs, but there are tools that use some of them (e.g., __doc__ holds the docstrings discussed in Chapter 15).

Also, observe that Y, a second instance made at the start of this series, still has an empty namespace dictionary at the end, even though X’s dictionary has been populated by assignments in methods. Again, each instance has an independent namespace dictionary, which starts out empty and can record completely different attributes than those recorded by the namespace dictionaries of other instances of the same class.

Because attributes are actually dictionary keys inside Python, there are really two ways to fetch and assign their values—by qualification, or by key indexing:

>>> X.data1, X.__dict__['data1']
('spam', 'spam')

>>> X.data3 = 'toast'
>>> X.__dict__
{'data2': 'eggs', 'data3': 'toast', 'data1': 'spam'}

>>> X.__dict__['data3'] = 'ham'
>>> X.data3
'ham'

This equivalence applies only to attributes actually attached to the instance, though. Because attribute fetch qualification also performs an inheritance search, it can access inherited attributes that namespace dictionary indexing cannot. The inherited attribute X.hello, for instance, cannot be accessed by X.__dict__['hello'].

Experiment with these special attributes on your own to get a better feel for how namespaces actually do their attribute business. Also try running these objects through the dir function we met in the prior two chapters—dir(X) is similar to X.__dict__.keys(), but dir sorts its list and includes some inherited and built-in attributes. Even if you will never use these in the kinds of programs you write, seeing that they are just normal dictionaries can help solidify namespaces in general.

The prior section demonstrated the special __class__ and __bases__ instance and class attributes, without really explaining why you might care about them. In short, these attributes allow you to inspect inheritance hierarchies within your own code. For example, they can be used to display a class tree, as in the following Python 3.X and 2.X example:

#!python
"""
classtree.py: Climb inheritance trees using namespace links,
displaying higher superclasses with indentation for height
"""

def classtree(cls, indent):
    print('.' * indent + cls.__name__)    # Print class name here
    for supercls in cls.__bases__:        # Recur to all superclasses
        classtree(supercls, indent+3)     # May visit super > once

def instancetree(inst):
    print('Tree of %s' % inst)            # Show instance
    classtree(inst.__class__, 3)          # Climb to its class

def selftest():
    class A:      pass
    class B(A):   pass
    class C(A):   pass
    class D(B,C): pass
    class E:      pass
    class F(D,E): pass
    instancetree(B())
    instancetree(F())

if __name__ == '__main__': selftest()

The classtree function in this script is recursive—it prints a class’s name using __name__, then climbs up to the superclasses by calling itself. This allows the function to traverse arbitrarily shaped class trees; the recursion climbs to the top, and stops at root superclasses that have empty __bases__ attributes. When using recursion, each active level of a function gets its own copy of the local scope; here, this means that cls and indent are different at each classtree level.

Most of this file is self-test code. When run standalone in Python 2.X, it builds an empty class tree, makes two instances from it, and prints their class tree structures:

C:\code> c:\python27\python classtree.py
Tree of <__main__.B instance at 0x00000000022C3A88>
...B
......A
Tree of <__main__.F instance at 0x00000000022C3A88>
...F
......D
.........B
............A
.........C
............A
......E

When run by Python 3.X, the tree includes the implied object superclass that is automatically added above standalone root (i.e., topmost) classes, because all classes are “new style” in 3.X—more on this change in Chapter 32:

C:\code> c:\python33\python classtree.py
Tree of <__main__.selftest.<locals>.B object at 0x00000000029216A0>
...B
......A
.........object
Tree of <__main__.selftest.<locals>.F object at 0x00000000029216A0>
...F
......D
.........B
............A
...............object
.........C
............A
...............object
......E
.........object

Here, indentation marked by periods is used to denote class tree height. Of course, we could improve on this output format, and perhaps even sketch it in a GUI display. Even as is, though, we can import these functions anywhere we want a quick display of a physical class tree:

C:\code> c:\python33\python
>>> class Emp: pass

>>> class Person(Emp): pass

>>> bob = Person()

>>> import classtree
>>> classtree.instancetree(bob)
Tree of <__main__.Person object at 0x000000000298B6D8>
...Person
......Emp
.........object

Regardless of whether you will ever code or use such tools, this example demonstrates one of the many ways that you can make use of special attributes that expose interpreter internals. You’ll see another when we code the lister.py general-purpose class display tools in Chapter 31’s section Multiple Inheritance: “Mix-in” Classes—there, we will extend this technique to also display attributes in each object in a class tree and function as a common superclass.

In the last part of this book, we’ll revisit such tools in the context of Python tool building at large, to code tools that implement attribute privacy, argument validation, and more. While not in every Python programmer’s job description, access to internals enables powerful development tools.