Python functions are more flexible than you might think. As we’ve seen in this part of the book, functions in Python are much more than code-generation specifications for a compiler—Python functions are full-blown objects, stored in pieces of memory all their own. As such, they can be freely passed around a program and called indirectly. They also support operations that have little to do with calls at all—attribute storage and annotation.
Because Python functions are objects, you can write programs that process them generically. Function objects may be assigned to other names, passed to other functions, embedded in data structures, returned from one function to another, and more, as if they were simple numbers or strings. Function objects also happen to support a special operation: they can be called by listing arguments in parentheses after a function expression. Still, functions belong to the same general category as other objects.
This is usually called a first-class object model; it’s ubiquitous in Python, and a necessary part of functional programming. We’ll explore this programming mode more fully in this and the next chapter; because its motif is founded on the notion of applying functions, functions must be treated as data.
We’ve seen some of these generic use cases for functions in earlier examples, but a quick review helps to underscore the object model. For example, there’s really nothing special about the name used in a def statement: it’s just a variable assigned in the current scope, as if it had appeared on the left of an = sign. After a def runs, the function name is simply a reference to an object—you can reassign that object to other names freely and call it through any reference:
>>>def echo(message):# Name echo assigned to function objectprint(message)>>>echo('Direct call')# Call object through original name Direct call >>>x = echo# Now x references the function too >>>x('Indirect call!')# Call object through name by adding () Indirect call!
Because arguments are passed by assigning objects, it’s just as easy to pass functions to other functions as arguments. The callee may then call the passed-in function just by adding arguments in parentheses:
>>>def indirect(func, arg):func(arg)# Call the passed-in object by adding () >>>indirect(echo, 'Argument call!')# Pass the function to another function Argument call!
You can even stuff function objects into data structures, as though they were integers or strings. The following, for example, embeds the function twice in a list of tuples, as a sort of actions table. Because Python compound types like these can contain any sort of object, there’s no special case here, either:
>>>schedule = [ (echo, 'Spam!'), (echo, 'Ham!') ]>>>for (func, arg) in schedule:func(arg)# Call functions embedded in containers Spam! Ham!
This code simply steps through the schedule list, calling the echo function with one argument each time through (notice the tuple-unpacking assignment in the for loop header, introduced in Chapter 13). As we saw in Chapter 17’s examples, functions can also be created and returned for use elsewhere—the closure created in this mode also retains state from the enclosing scope:
>>>def make(label):# Make a function but don't call itdef echo(message):print(label + ':' + message)return echo>>>F = make('Spam')# Label in enclosing scope is retained >>>F('Ham!')# Call the function that make returned Spam:Ham! >>>F('Eggs!')Spam:Eggs!
Python’s universal first-class object model and lack of type declarations make for an incredibly flexible programming language.
Because they are objects, we can also process functions with normal object tools. In fact, functions are more flexible than you might expect. For instance, once we make a function, we can call it as usual:
>>>def func(a):b = 'spam'return b * a>>>func(8)'spamspamspamspamspamspamspamspam'
But the call expression is just one operation defined to work on function objects. We can also inspect their attributes generically (the following is run in Python 3.3, but 2.X results are similar):
>>>func.__name__'func' >>>dir(func)['__annotations__', '__call__', '__class__', '__closure__', '__code__',...more omitted: 34 total...'__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__']
Introspection tools allow us to explore implementation details too—functions have attached code objects, for example, which provide details on aspects such as the functions’ local variables and arguments:
>>>func.__code__<code object func at 0x00000000021A6030, file "<stdin>", line 1> >>>dir(func.__code__)['__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__',...more omitted: 37 total...'co_argcount', 'co_cellvars', 'co_code', 'co_consts', 'co_filename', 'co_firstlineno', 'co_flags', 'co_freevars', 'co_kwonlyargcount', 'co_lnotab', 'co_name', 'co_names', 'co_nlocals', 'co_stacksize', 'co_varnames'] >>>func.__code__.co_varnames('a', 'b') >>>func.__code__.co_argcount1
Tool writers can make use of such information to manage functions (in fact, we will too in Chapter 39, to implement validation of function arguments in decorators).
Function objects are not limited to the system-defined attributes listed in the prior section, though. As we learned in Chapter 17, it’s been possible to attach arbitrary user-defined attributes to them as well since Python 2.1:
>>>func<function func at 0x000000000296A1E0> >>>func.count = 0>>>func.count += 1>>>func.count1 >>>func.handles = 'Button-Press'>>>func.handles'Button-Press' >>>dir(func)['__annotations__', '__call__', '__class__', '__closure__', '__code__',...and more: in 3.X all others have double underscores so your names won't clash...__str__', '__subclasshook__', 'count', 'handles']
Python’s own implementation-related data stored on functions follows naming conventions that prevent them from clashing with the more arbitrary attribute names you might assign yourself. In 3.X, all function internals’ names have leading and trailing double underscores (“__X__”); 2.X follows the same scheme, but also assigns some names that begin with “func_X”:
c:\code>py −3>>>def f(): pass>>>dir(f)...run on your own to see...>>>len(dir(f))34 >>>[x for x in dir(f) if not x.startswith('__')][] c:\code>py −2>>>def f(): pass>>>dir(f)...run on your own to see...>>>len(dir(f))31 >>>[x for x in dir(f) if not x.startswith('__')]['func_closure', 'func_code', 'func_defaults', 'func_dict', 'func_doc', 'func_globals', 'func_name']
If you’re careful not to name attributes the same way, you can safely use the function’s namespace as though it were your own namespace or scope.
As we saw in that chapter, such attributes can be used to attach state information to function objects directly, instead of using other techniques such as globals, nonlocals, and classes. Unlike nonlocals, such attributes are accessible anywhere the function itself is, even from outside its code.
In a sense, this is also a way to emulate “static locals” in other languages—variables whose names are local to a function, but whose values are retained after a function exits. Attributes are related to objects instead of scopes (and must be referenced through the function name within its code), but the net effect is similar.
Moreover, as we learned in Chapter 17, when attributes are attached to functions generated by other factory functions, they also support multiple copy, per-call, and writeable state retention, much like nonlocal closures and class instance attributes.
In Python 3.X (but not 2.X), it’s also possible to attach annotation information—arbitrary user-defined data about a function’s arguments and result—to a function object. Python provides special syntax for specifying annotations, but it doesn’t do anything with them itself; annotations are completely optional, and when present are simply attached to the function object’s __annotations__ attribute for use by other tools. For instance, such a tool might use annotations in the context of error testing.
We met Python 3.X’s keyword-only arguments in the preceding chapter; annotations generalize function header syntax further. Consider the following nonannotated function, which is coded with three arguments and returns a result:
>>>def func(a, b, c):return a + b + c>>>func(1, 2, 3)6
Syntactically, function annotations are coded in def header lines, as arbitrary expressions associated with arguments and return values. For arguments, they appear after a colon immediately following the argument’s name; for return values, they are written after a -> following the arguments list. This code, for example, annotates all three of the prior function’s arguments, as well as its return value:
>>>def func(a: 'spam', b: (1, 10), c: float) -> int:return a + b + c>>>func(1, 2, 3)6
Calls to an annotated function work as usual, but when annotations are present Python collects them in a dictionary and attaches it to the function object itself. Argument names become keys, the return value annotation is stored under key “return” if coded (which suffices because this reserved word can’t be used as an argument name), and the values of annotation keys are assigned to the results of the annotation expressions:
>>> func.__annotations__
{'c': <class 'float'>, 'b': (1, 10), 'a': 'spam', 'return': <class 'int'>}
Because they are just Python objects attached to a Python object, annotations are straightforward to process. The following annotates just two of three arguments and steps through the attached annotations generically:
>>>def func(a: 'spam', b, c: 99):return a + b + c>>>func(1, 2, 3)6 >>>func.__annotations__{'c': 99, 'a': 'spam'} >>>for arg in func.__annotations__:print(arg, '=>', func.__annotations__[arg])c => 99 a => spam
There are two fine points to note here. First, you can still use defaults for arguments if you code annotations—the annotation (and its : character) appear before the default (and its = character). In the following, for example, a: 'spam' = 4 means that argument a defaults to 4 and is annotated with the string 'spam':
>>>def func(a: 'spam' = 4, b: (1, 10) = 5, c: float = 6) -> int:return a + b + c>>>func(1, 2, 3)6 >>>func()# 4 + 5 + 6 (all defaults) 15 >>>func(1, c=10)# 1 + 5 + 10 (keywords work normally) 16 >>>func.__annotations__{'c': <class 'float'>, 'b': (1, 10), 'a': 'spam', 'return': <class 'int'>}
Second, note that the blank spaces in the prior example are all optional—you can use spaces between components in function headers or not, but omitting them might degrade your code’s readability to some observers (and probably improve it to others!):
>>>def func(a:'spam'=4, b:(1,10)=5, c:float=6)->int:return a + b + c>>>func(1, 2)# 1 + 2 + 6 9 >>>func.__annotations__{'c': <class 'float'>, 'b': (1, 10), 'a': 'spam', 'return': <class 'int'>}
Annotations are a new feature in 3.X, and some of their potential uses remain to be uncovered. It’s easy to imagine annotations being used to specify constraints for argument types or values, though, and larger APIs might use this feature as a way to register function interface information.
In fact, we’ll see a potential application in Chapter 39, where we’ll look at annotations as an alternative to function decorator arguments—a more general concept in which information is coded outside the function header and so is not limited to a single role. Like Python itself, annotation is a tool whose roles are shaped by your imagination.
Finally, note that annotations work only in def statements, not lambda expressions, because lambda’s syntax already limits the utility of the functions it defines. Coincidentally, this brings us to our next topic.