Why nonlocal? State Retention Options

As we saw in the prior section, the following code allows state to be retained and modified in an enclosing scope. Each call to tester creates a self-contained package of changeable information, whose names do not clash with any other part of the program:

>>> def tester(start):
        state = start                  # Each call gets its own state
        def nested(label):
            nonlocal state             # Remembers state in enclosing scope
            print(label, state)
            state += 1                 # Allowed to change it if nonlocal
        return nested

>>> F = tester(0)
>>> F('spam')                          # State visible within closure only
spam 0
>>> F.state
AttributeError: 'function' object has no attribute 'state'

We need to declare variables nonlocal only if they must be changed (other enclosing scope name references are automatically retained as usual), and nonlocal names are still not visible outside the enclosing function.

Unfortunately, this code works in Python 3.X only. If you are using Python 2.X, other options are available, depending on your goals. The next three sections present some alternatives. Some of the code in these sections uses tools we haven’t covered yet and is intended partially as preview, but we’ll keep the examples simple here so that you can compare and contrast along the way.

One common prescription for achieving the nonlocal effect in 2.X and earlier is to simply move the state out to the global scope (the enclosing module):

>>> def tester(start):
        global state                   # Move it out to the module to change it
        state = start                  # global allows changes in module scope
        def nested(label):
            global state
            print(label, state)
            state += 1
        return nested

>>> F = tester(0)
>>> F('spam')                          # Each call increments shared global state
spam 0
>>> F('eggs')
eggs 1

This works in this case, but it requires global declarations in both functions and is prone to name collisions in the global scope (what if “state” is already being used?). A worse, and more subtle, problem is that it only allows for a single shared copy of the state information in the module scope—if we call tester again, we’ll wind up resetting the module’s state variable, such that prior calls will see their state overwritten:

>>> G = tester(42)                     # Resets state's single copy in global scope
>>> G('toast')
toast 42

>>> G('bacon')
bacon 43

>>> F('ham')                           # But my counter has been overwritten!
ham 44

As shown earlier, when you are using nonlocal and nested function closures instead of global, each call to tester remembers its own unique copy of the state object.

The other prescription for changeable state information in 2.X and earlier is to use classes with attributes to make state information access more explicit than the implicit magic of scope lookup rules. As an added benefit, each instance of a class gets a fresh copy of the state information, as a natural byproduct of Python’s object model. Classes also support inheritance, multiple behaviors, and other tools.

We haven’t explored classes in detail yet, but as a brief preview for comparison, the following is a reformulation of the earlier tester/nested functions as a class, which records state in objects explicitly as they are created. To make sense of this code, you need to know that a def within a class like this works exactly like a normal def, except that the function’s self argument automatically receives the implied subject of the call (an instance object created by calling the class itself). The function named __init__ is run automatically when the class is called:

>>> class tester:                          # Class-based alternative (see Part VI)
        def __init__(self, start):         # On object construction,
            self.state = start             # save state explicitly in new object
        def nested(self, label):
            print(label, self.state)       # Reference state explicitly
            self.state += 1                # Changes are always allowed

>>> F = tester(0)                          # Create instance, invoke __init__
>>> F.nested('spam')                       # F is passed to self
spam 0
>>> F.nested('ham')
ham 1

In classes, we save every attribute explicitly, whether it’s changed or just referenced, and they are available outside the class. As for nested functions and nonlocal, the class alternative supports multiple copies of the retained data:

>>> G = tester(42)                         # Each instance gets new copy of state
>>> G.nested('toast')                      # Changing one does not impact others
toast 42
>>> G.nested('bacon')
bacon 43

>>> F.nested('eggs')                       # F's state is where it left off
eggs 2
>>> F.state                                # State may be accessed outside class
3

With just slightly more magic—which we’ll delve into later in this book—we could also make our class objects look like callable functions using operator overloading. __call__ intercepts direct calls on an instance, so we don’t need to call a named method:

>>> class tester:
        def __init__(self, start):
            self.state = start
        def __call__(self, label):         # Intercept direct instance calls
            print(label, self.state)       # So .nested() not required
            self.state += 1

>>> H = tester(99)
>>> H('juice')                             # Invokes __call__
juice 99
>>> H('pancakes')
pancakes 100

Don’t sweat the details in this code too much at this point in the book; it’s mostly a preview, intended for general comparison to closures only. We’ll explore classes in depth in Part VI, and will look at specific operator overloading tools like __call__ in Chapter 30. The point to notice here is that classes can make state information more obvious, by leveraging explicit attribute assignment instead of implicit scope lookups. In addition, class attributes are always changeable and don’t require a nonlocal statement, and classes are designed to scale up to implementing richer objects with many attributes and behaviors.

While using classes for state information is generally a good rule of thumb to follow, they might also be overkill in cases like this, where state is a single counter. Such trivial state cases are more common than you might think; in such contexts, nested defs are sometimes more lightweight than coding classes, especially if you’re not familiar with OOP yet. Moreover, there are some scenarios in which nested defs may actually work better than classes—stay tuned for the description of method decorators in Chapter 39 for an example that is far beyond this chapter’s already well-stretched scope!

As a portable and often simpler state-retention option, we can also sometimes achieve the same effect as nonlocals with function attributes—user-defined names attached to functions directly. When you attach user-defined attributes to nested functions generated by enclosing factory functions, they can also serve as per-call, multiple copy, and writeable state, just like nonlocal scope closures and class attributes. Such user-defined attribute names won’t clash with names Python creates itself, and as for nonlocal, need be used only for state variables that must be changed; other scope references are retained and work normally.

Crucially, this scheme is portable—like classes, but unlike nonlocal, function attributes work in both Python 3.X and 2.X. In fact, they’ve been available since 2.1, much longer than 3.X’s nonlocal. Because factory functions make a new function on each call anyhow, this does not require extra objects—the new function’s attributes become per-call state in much the same way as nonlocals, and are similarly associated with the generated function in memory.

Moreover, function attributes allow state variables to be accessed outside the nested function, like class attributes; with nonlocal, state variables can be seen directly only within the nested def. If you need to access a call counter externally, it’s a simple function attribute fetch in this model.

Here’s a final version of our example based on this technique—it replaces a nonlocal with an attribute attached to the nested function. This scheme may not seem as intuitive to some at first glance; you access state though the function’s name instead of as simple variables, and must initialize after the nested def. Still, it’s far more portable, allows state to be accessed externally, and saves a line by not requiring a nonlocal declaration:

>>> def tester(start):
        def nested(label):
            print(label, nested.state)     # nested is in enclosing scope
            nested.state += 1              # Change attr, not nested itself
        nested.state = start               # Initial state after func defined
        return nested

>>> F = tester(0)
>>> F('spam')                              # F is a 'nested' with state attached
spam 0
>>> F('ham')
ham 1
>>> F.state                                # Can access state outside functions too
2

Because each call to the outer function produces a new nested function object, this scheme supports multiple copy per-call changeable data just like nonlocal closures and classes—a usage mode that global variables cannot provide:

>>> G = tester(42)                         # G has own state, doesn't overwrite F's
>>> G('eggs')
eggs 42
>>> F('ham')
ham 2

>>> F.state                                # State is accessible and per-call
3
>>> G.state
43
>>> F is G                                 # Different function objects
False

This code relies on the fact that the function name nested is a local variable in the tester scope enclosing nested; as such, it can be referenced freely inside nested. This code also relies on the fact that changing an object in place is not an assignment to a name; when it increments nested.state, it is changing part of the object nested references, not the name nested itself. Because we’re not really assigning a name in the enclosing scope, no nonlocal declaration is required.

Function attributes are supported in both Python 3.X and 2.X; we’ll explore them further in Chapter 19. Importantly, we’ll see there that Python uses naming conventions in both 2.X and 3.X that ensure that the arbitrary names you assign as function attributes won’t clash with names related to internal implementation, making the namespace equivalent to a scope. Subjective factors aside, function attributes’ utility does overlap with the newer nonlocal in 3.X, making the latter technically redundant and far less portable.

def tester(start):
    def nested(label):
        print(label, state[0])             # Leverage in-place mutable change
        state[0] += 1                      # Extra syntax, deep magic?
    state = [start]
    return nested

import builtins

def makeopen(id):
    original = builtins.open
    def custom(*kargs, **pargs):
        print('Custom open call %r:' % id , kargs, pargs)
        return original(*kargs, **pargs)
    builtins.open = custom

>>> F = open('script2.py')           # Call built-in open in builtins
>>> F.read()
'import sys\nprint(sys.path)\nx = 2\nprint(x ** 32)\n'

>>> from makeopen import makeopen    # Import open resetter function
>>> makeopen('spam')                 # Custom open calls built-in open

>>> F = open('script2.py')           # Call custom open in builtins
Custom open call 'spam': ('script2.py',) {}
>>> F.read()
'import sys\nprint(sys.path)\nx = 2\nprint(x ** 32)\n'

>>> makeopen('eggs')                 # Nested customizers work too!
>>> F = open('script2.py')           # Because each retains own state
Custom open call 'eggs': ('script2.py',) {}
Custom open call 'spam': ('script2.py',) {}
>>> F.read()
'import sys\nprint(sys.path)\nx = 2\nprint(x ** 32)\n'

import builtins

class makeopen:                # See Part VI: call catches self()
    def __init__(self, id):
        self.id = id
        self.original = builtins.open
        builtins.open = self
    def __call__(self, *kargs, **pargs):
        print('Custom open call %r:' % self.id, kargs, pargs)
        return self.original(*kargs, **pargs)