Function Gotchas

As you know, Python classifies names assigned in a function as locals by default; they live in the function’s scope and exist only while the function is running. What you may not realize is that Python detects locals statically, when it compiles the def’s code, rather than by noticing assignments as they happen at runtime. This leads to one of the most common oddities posted on the Python newsgroup by beginners.

Normally, a name that isn’t assigned in a function is looked up in the enclosing module:

>>> X = 99

>>> def selector():       # X used but not assigned
        print(X)          # X found in global scope

>>> selector()
99

Here, the X in the function resolves to the X in the module. But watch what happens if you add an assignment to X after the reference:

>>> def selector():
        print(X)          # Does not yet exist!
        X = 88            # X classified as a local name (everywhere)
                          # Can also happen for "import X", "def X"...
>>> selector()
UnboundLocalError: local variable 'X' referenced before assignment

You get the name usage error shown here, but the reason is subtle. Python reads and compiles this code when it’s typed interactively or imported from a module. While compiling, Python sees the assignment to X and decides that X will be a local name everywhere in the function. But when the function is actually run, because the assignment hasn’t yet happened when the print executes, Python says you’re using an undefined name. According to its name rules, it should say this; the local X is used before being assigned. In fact, any assignment in a function body makes a name local. Imports, =, nested defs, nested classes, and so on are all susceptible to this behavior.

The problem occurs because assigned names are treated as locals everywhere in a function, not just after the statements where they’re assigned. Really, the previous example is ambiguous: was the intention to print the global X and create a local X, or is this a real programming error? Because Python treats X as a local everywhere, it’s seen as an error; if you mean to print the global X, you need to declare it in a global statement:

>>> def selector():
        global X                # Force X to be global (everywhere)
        print(X)
        X = 88

>>> selector()
99

Remember, though, that this means the assignment also changes the global X, not a local X. Within a function, you can’t use both local and global versions of the same simple name. If you really meant to print the global and then set a local of the same name, you’d need to import the enclosing module and use module attribute notation to get to the global version:

>>> X = 99
>>> def selector():
        import __main__         # Import enclosing module
        print(__main__.X)       # Qualify to get to global version of name
        X = 88                  # Unqualified X classified as local
        print(X)                # Prints local version of name

>>> selector()
99
88

Qualification (the .X part) fetches a value from a namespace object. The interactive namespace is a module called __main__, so __main__.X reaches the global version of X. If that isn’t clear, check out Chapter 17.

In recent versions Python has improved on this story somewhat by issuing for this case the more specific “unbound local” error message shown in the example listing (it used to simply raise a generic name error); this gotcha is still present in general, though.

As noted briefly in Chapter 17 and Chapter 18, mutable values for default arguments can retain state between calls, though this is often unexpected. In general, default argument values are evaluated and saved once when a def statement is run, not each time the resulting function is later called. Internally, Python saves one object per default argument attached to the function itself.

That’s usually what you want—because defaults are evaluated at def time, it lets you save values from the enclosing scope, if needed (functions defined within loops by factories may even depend on this behavior—see ahead). But because a default retains an object between calls, you have to be careful about changing mutable defaults. For instance, the following function uses an empty list as a default value, and then changes it in place each time the function is called:

>>> def saver(x=[]):               # Saves away a list object
        x.append(1)                # Changes same object each time!
        print(x)

>>> saver([2])                     # Default not used
[2, 1]
>>> saver()                        # Default used
[1]
>>> saver()                        # Grows on each call!
[1, 1]
>>> saver()
[1, 1, 1]

Some see this behavior as a feature—because mutable default arguments retain their state between function calls, they can serve some of the same roles as static local function variables in the C language. In a sense, they work much like global variables, but their names are local to the functions and so will not clash with names elsewhere in a program.

To other observers, though, this seems like a gotcha, especially the first time they run into it. There are better ways to retain state between calls in Python (e.g., using the nested scope closures we met in this part and the classes we will study in Part VI).

Moreover, mutable defaults are tricky to remember (and to understand at all). They depend upon the timing of default object construction. In the prior example, there is just one list object for the default value—the one created when the def is executed. You don’t get a new list every time the function is called, so the list grows with each new append; it is not reset to empty on each call.

If that’s not the behavior you want, simply make a copy of the default at the start of the function body, or move the default value expression into the function body. As long as the value resides in code that’s actually executed each time the function runs, you’ll get a new object each time through:

>>> def saver(x=None):
        if x is None:             # No argument passed?
            x = []                # Run code to make a new list each time
        x.append(1)               # Changes new list object
        print(x)

>>> saver([2])
[2, 1]
>>> saver()                       # Doesn't grow here
[1]
>>> saver()
[1]

By the way, the if statement in this example could almost be replaced by the assignment x = x or [], which takes advantage of the fact that Python’s or returns one of its operand objects: if no argument was passed, x would default to None, so the or would return the new empty list on the right.

However, this isn’t exactly the same. If an empty list were passed in, the or expression would cause the function to extend and return a newly created list, rather than extending and returning the passed-in list like the if version. (The expression becomes [] or [], which evaluates to the new empty list on the right; see the section “Truth Tests” if you don’t recall why.) Real program requirements may call for either behavior.

Today, another way to achieve the value retention effect of mutable defaults in a possibly less confusing way is to use the function attributes we discussed in Chapter 19:

>>> def saver():
        saver.x.append(1)
        print(saver.x)

>>> saver.x = []
>>> saver()
[1]
>>> saver()
[1, 1]
>>> saver()
[1, 1, 1]

The function name is global to the function itself, but it need not be declared because it isn’t changed directly within the function. This isn’t used in exactly the same way, but when coded like this, the attachment of an object to the function is much more explicit (and arguably less magical).

In Python functions, return (and yield) statements are optional. When a function doesn’t return a value explicitly, the function exits when control falls off the end of the function body. Technically, all functions return a value; if you don’t provide a return statement, your function returns the None object automatically:

>>> def proc(x):
        print(x)                 # No return is a None return

>>> x = proc('testing 123...')
testing 123...
>>> print(x)
None

Functions such as this without a return are Python’s equivalent of what are called “procedures” in some languages. They’re usually invoked as statements, and the None results are ignored, as they do their business without computing a useful result.

This is worth knowing, because Python won’t tell you if you try to use the result of a function that doesn’t return one. As we noted in Chapter 11, for instance, assigning the result of a list append method won’t raise an error, but you’ll get back None, not the modified list:

>>> list = [1, 2, 3]
>>> list = list.append(4)        # append is a "procedure"
>>> print(list)                  # append changes list in place
None

Chapter 15’s section Common Coding Gotchas discusses this more broadly. In general, any functions that do their business as a side effect are usually designed to be run as statements, not expressions.

Here are two additional function-related gotchas—mostly reviews, but common enough to reiterate.