Scopes and Nested Functions

Factory functions (a.k.a. closures) are sometimes used by programs that need to generate event handlers on the fly in response to conditions at runtime. For instance, imagine a GUI that must define actions according to user inputs that cannot be anticipated when the GUI is built. In such cases, we need a function that creates and returns another function, with information that may vary per function made.

To illustrate this in simple terms, consider the following function, typed at the interactive prompt (and shown here without the “...” continuation-line prompts, per the presentation note ahead):

>>> def maker(N):
        def action(X):                    # Make and return action
            return X ** N                 # action retains N from enclosing scope
        return action

This defines an outer function that simply generates and returns a nested function, without calling it—maker makes action, but simply returns action without running it. If we call the outer function:

>>> f = maker(2)                          # Pass 2 to argument N
>>> f
<function maker.<locals>.action at 0x0000000002A4A158>

what we get back is a reference to the generated nested function—the one created when the nested def runs. If we now call what we got back from the outer function:

>>> f(3)                                  # Pass 3 to X, N remembers 2: 3 ** 2
9
>>> f(4)                                  # 4 ** 2
16

we invoke the nested function—the one called action within maker. In other words, we’re calling the nested function that maker created and passed back.

Perhaps the most unusual part of this, though, is that the nested function remembers integer 2, the value of the variable N in maker, even though maker has returned and exited by the time we call action. In effect, N from the enclosing local scope is retained as state information attached to the generated action, which is why we get back its argument squared when it is later called.

Just as important, if we now call the outer function again, we get back a new nested function with different state information attached. That is, we get the argument cubed instead of squared when calling the new function, but the original still squares as before:

>>> g = maker(3)                          # g remembers 3, f remembers 2
>>> g(4)                                  # 4 ** 3
64
>>> f(4)                                  # 4 ** 2
16

This works because each call to a factory function like this gets its own set of state information. In our case, the function we assign to name g remembers 3, and f remembers 2, because each has its own state information retained by the variable N in maker.

This is a somewhat advanced technique that you may not see very often in most code, and may be popular among programmers with backgrounds in functional programming languages. On the other hand, enclosing scopes are often employed by the lambda function-creation expressions we’ll expand on later in this chapter—because they are expressions, they are almost always nested within a def. For example, a lambda would serve in place of a def in our example:

>>> def maker(N):
        return lambda X: X ** N           # lambda functions retain state too

>>> h = maker(3)
>>> h(4)                                  # 4 ** 3 again
64

For a more tangible example of closures at work, see the upcoming sidebar Why You Will Care: Customizing open. It uses similar techniques to store information for later use in an enclosing scope.

To some, classes, described in full in Part VI of this book, may seem better at state retention like this, because they make their memory more explicit with attribute assignments. Classes also directly support additional tools that closure functions do not, such as customization by inheritance and operator overloading, and more naturally implement multiple behaviors in the form of methods. Because of such distinctions, classes may be better at implementing more complete objects.

Still, closure functions often provide a lighter-weight and viable alternative when retaining state is the only goal. They provide for per-call localized storage for data required by a single nested function. This is especially true when we add the 3.X nonlocal statement described ahead to allow enclosing scope state changes (in 2.X, enclosing scopes are read-only, and so have more limited uses).

From a broader perspective, there are multiple ways for Python functions to retain state between calls. Although the values of normal local variables go away when a function returns, values can be retained from call to call in global variables; in class instance attributes; in the enclosing scope references we’ve met here; and in argument defaults and function attributes. Some might include mutable default arguments to this list too (though others may wish they didn’t).

We’ll preview class-based alternatives and meet function attributes later in this chapter, and get the full story on arguments and defaults in Chapter 18. To help us judge how defaults compete on state retention, though, the next section gives enough of an introduction to get us started.

def f1():
    x = 88
    def f2(x=x):                # Remember enclosing scope X with defaults
        print(x)
    f2()

f1()                            # Prints 88

>>> def f1():
        x = 88                  # Pass x along instead of nesting
        f2(x)                   # Forward reference OK

>>> def f2(x):
        print(x)                # Flat is still often better than nested!

>>> f1()
88

Although they see increasing use in defs these days, you may be more likely to care about nested function scopes when you start coding or reading lambda expressions. We’ve met lambda briefly and won’t cover it in depth until Chapter 19, but in short, it’s an expression that generates a new function to be called later, much like a def statement. Because it’s an expression, though, it can be used in places that def cannot, such as within list and dictionary literals.

Like a def, a lambda expression also introduces a new local scope for the function it creates. Thanks to the enclosing scopes lookup layer, lambdas can see all the variables that live in the functions in which they are coded. Thus, the following code—a variation on the factory we saw earlier—works, but only because the nested scope rules are applied:

def func():
    x = 4
    action = (lambda n: x ** n)          # x remembered from enclosing def
    return action

x = func()
print(x(2))                              # Prints 16, 4 ** 2

Prior to the introduction of nested function scopes, programmers used defaults to pass values from an enclosing scope into lambdas, just as for defs. For instance, the following works on all Pythons:

def func():
    x = 4
    action = (lambda n, x=x: x ** n)     # Pass x in manually
    return action

Because lambdas are expressions, they naturally (and even normally) nest inside enclosing defs. Hence, they were perhaps the biggest initial beneficiaries of the addition of enclosing function scopes in the lookup rules; in most cases, it is no longer necessary to pass values into lambdas with defaults.

There is one notable exception to the rule I just gave (and a reason why I’ve shown you the otherwise dated default argument technique we just saw): if a lambda or def defined within a function is nested inside a loop, and the nested function references an enclosing scope variable that is changed by that loop, all functions generated within the loop will have the same value—the value the referenced variable had in the last loop iteration. In such cases, you must still use defaults to save the variable’s current value instead.

This may seem a fairly obscure case, but it can come up in practice more often than you may think, especially in code that generates callback handler functions for a number of widgets in a GUI—for instance, handlers for button-clicks for all the buttons in a row. If these are created in a loop, you may need to be careful to save state with defaults, or all your buttons’ callbacks may wind up doing the same thing.

Here’s an illustration of this phenomenon reduced to simple code: the following attempts to build up a list of functions that each remember the current variable i from the enclosing scope:

>>> def makeActions():
        acts = []
        for i in range(5):                       # Tries to remember each i
            acts.append(lambda x: i ** x)        # But all remember same last i!
        return acts

>>> acts = makeActions()
>>> acts[0]
<function makeActions.<locals>.<lambda> at 0x0000000002A4A400>

This doesn’t quite work, though—because the enclosing scope variable is looked up when the nested functions are later called, they all effectively remember the same value: the value the loop variable had on the last loop iteration. That is, when we pass a power argument of 2 in each of the following calls, we get back 4 to the power of 2 for each function in the list, because i is the same in all of them—4:

>>> acts[0](2)                                   # All are 4 ** 2, 4=value of last i
16
>>> acts[1](2)                                   # This should be 1 ** 2 (1)
16
>>> acts[2](2)                                   # This should be 2 ** 2 (4)
16
>>> acts[4](2)                                   # Only this should be 4 ** 2 (16)
16

This is the one case where we still have to explicitly retain enclosing scope values with default arguments, rather than enclosing scope references. That is, to make this sort of code work, we must pass in the current value of the enclosing scope’s variable with a default. Because defaults are evaluated when the nested function is created (not when it’s later called), each remembers its own value for i:

>>> def makeActions():
        acts = []
        for i in range(5):                       # Use defaults instead
            acts.append(lambda x, i=i: i ** x)   # Remember current i
        return acts

>>> acts = makeActions()
>>> acts[0](2)                                   # 0 ** 2
0
>>> acts[1](2)                                   # 1 ** 2
1
>>> acts[2](2)                                   # 2 ** 2
4
>>> acts[4](2)                                   # 4 ** 2
16

This seems an implementation artifact that is prone to change, and may become more important as you start writing larger programs. We’ll talk more about defaults in Chapter 18 and lambdas in Chapter 19, so you may also want to return and review this section later.^[37]

Before ending this discussion, we should note that scopes may nest arbitrarily, but only enclosing function def statements (not classes, described in Part VI) are searched when names are referenced:

>>> def f1():
        x = 99
        def f2():
            def f3():
                print(x)        # Found in f1's local scope!
            f3()
        f2()

>>> f1()
99

Python will search the local scopes of all enclosing defs, from inner to outer, after the referencing function’s local scope and before the module’s global scope or built-ins. However, this sort of code is even less likely to pop up in practice. Again, in Python, we say flat is better than nested, and this still holds generally true even with the addition of nested scope closures. Except in limited contexts, your life (and the lives of your coworkers) will generally be better if you minimize nested function definitions.