Special Argument-Matching Modes

>>> def f(a, b, c): print(a, b, c)

>>> f(1, 2, 3)
1 2 3

In Python, though, you can be more specific about what goes where when you call a function. Keyword arguments allow us to match by name, instead of by position. Using the same function:

>>> f(c=3, b=2, a=1)
1 2 3

The c=3 in this call, for example, means send 3 to the argument named c. More formally, Python matches the name c in the call to the argument named c in the function definition’s header, and then passes the value 3 to that argument. The net effect of this call is the same as that of the prior call, but notice that the left-to-right order of the arguments no longer matters when keywords are used because arguments are matched by name, not by position. It’s even possible to combine positional and keyword arguments in a single call. In this case, all positionals are matched first from left to right in the header, before keywords are matched by name:

>>> f(1, c=3, b=2)            # a gets 1 by position, b and c passed by name
1 2 3

When most people see this the first time, they wonder why one would use such a tool. Keywords typically have two roles in Python. First, they make your calls a bit more self-documenting (assuming that you use better argument names than a, b, and c!). For example, a call of this form:

func(name='Bob', age=40, job='dev')

is much more meaningful than a call with three naked values separated by commas, especially in larger programs—the keywords serve as labels for the data in the call. The second major use of keywords occurs in conjunction with defaults, which we turn to next.

We talked about defaults in brief earlier, when discussing nested function scopes. In short, defaults allow us to make selected function arguments optional; if not passed a value, the argument is assigned its default before the function runs. For example, here is a function that requires one argument and defaults two:

>>> def f(a, b=2, c=3): print(a, b, c)          # a required, b and c optional

When we call this function, we must provide a value for a, either by position or by keyword; however, providing values for b and c is optional. If we don’t pass values to b and c, they default to 2 and 3, respectively:

>>> f(1)                   # Use defaults
1 2 3
>>> f(a=1)
1 2 3

If we pass two values, only c gets its default, and with three values, no defaults are used:

>>> f(1, 4)                # Override defaults
1 4 3
>>> f(1, 4, 5)
1 4 5

Finally, here is how the keyword and default features interact. Because they subvert the normal left-to-right positional mapping, keywords allow us to essentially skip over arguments with defaults:

>>> f(1, c=6)              # Choose defaults
1 2 6

Here, a gets 1 by position, c gets 6 by keyword, and b, in between, defaults to 2.

Be careful not to confuse the special name=value syntax in a function header and a function call; in the call it means a match-by-name keyword argument, while in the header it specifies a default for an optional argument. In both cases, this is not an assignment statement (despite its appearance); it is special syntax for these two contexts, which modifies the default argument-matching mechanics.

Here is a slightly larger example that demonstrates keywords and defaults in action. In the following, the caller must always pass at least two arguments (to match spam and eggs), but the other two are optional. If they are omitted, Python assigns toast and ham to the defaults specified in the header:

def func(spam, eggs, toast=0, ham=0):   # First 2 required
    print((spam, eggs, toast, ham))

func(1, 2)                              # Output: (1, 2, 0, 0)
func(1, ham=1, eggs=0)                  # Output: (1, 0, 0, 1)
func(spam=1, eggs=0)                    # Output: (1, 0, 0, 0)
func(toast=1, eggs=2, spam=3)           # Output: (3, 2, 1, 0)
func(1, 2, 3, 4)                        # Output: (1, 2, 3, 4)

Notice again that when keyword arguments are used in the call, the order in which the arguments are listed doesn’t matter; Python matches by name, not by position. The caller must supply values for spam and eggs, but they can be matched by position or by name. Again, keep in mind that the form name=value means different things in the call and the def: a keyword in the call and a default in the header.

The first use, in the function definition, collects unmatched positional arguments into a tuple:

>>> def f(*args): print(args)

When this function is called, Python collects all the positional arguments into a new tuple and assigns the variable args to that tuple. Because it is a normal tuple object, it can be indexed, stepped through with a for loop, and so on:

>>> f()
()
>>> f(1)
(1,)
>>> f(1, 2, 3, 4)
(1, 2, 3, 4)

The ** feature is similar, but it only works for keyword arguments—it collects them into a new dictionary, which can then be processed with normal dictionary tools. In a sense, the ** form allows you to convert from keywords to dictionaries, which you can then step through with keys calls, dictionary iterators, and the like (this is roughly what the dict call does when passed keywords, but it returns the new dictionary):

>>> def f(**args): print(args)

>>> f()
{}
>>> f(a=1, b=2)
{'a': 1, 'b': 2}

Finally, function headers can combine normal arguments, the *, and the ** to implement wildly flexible call signatures. For instance, in the following, 1 is passed to a by position, 2 and 3 are collected into the pargs positional tuple, and x and y wind up in the kargs keyword dictionary:

>>> def f(a, *pargs, **kargs): print(a, pargs, kargs)

>>> f(1, 2, 3, x=1, y=2)
1 (2, 3) {'y': 2, 'x': 1}

Such code is rare, but shows up in functions that need to support multiple call patterns (for backward compatibility, for instance). In fact, these features can be combined in even more complex ways that may seem ambiguous at first glance—an idea we will revisit later in this chapter. First, though, let’s see what happens when * and ** are coded in function calls instead of definitions.

In all recent Python releases, we can use the * syntax when we call a function, too. In this context, its meaning is the inverse of its meaning in the function definition—it unpacks a collection of arguments, rather than building a collection of arguments. For example, we can pass four arguments to a function in a tuple and let Python unpack them into individual arguments:

>>> def func(a, b, c, d): print(a, b, c, d)

>>> args = (1, 2)
>>> args += (3, 4)
>>> func(*args)                            # Same as func(1, 2, 3, 4)
1 2 3 4

Similarly, the ** syntax in a function call unpacks a dictionary of key/value pairs into separate keyword arguments:

>>> args = {'a': 1, 'b': 2, 'c': 3}
>>> args['d'] = 4
>>> func(**args)                           # Same as func(a=1, b=2, c=3, d=4)
1 2 3 4

Again, we can combine normal, positional, and keyword arguments in the call in very flexible ways:

>>> func(*(1, 2), **{'d': 4, 'c': 3})      # Same as func(1, 2, d=4, c=3)
1 2 3 4
>>> func(1, *(2, 3), **{'d': 4})           # Same as func(1, 2, 3, d=4)
1 2 3 4
>>> func(1, c=3, *(2,), **{'d': 4})        # Same as func(1, 2, c=3, d=4)
1 2 3 4
>>> func(1, *(2, 3), d=4)                  # Same as func(1, 2, 3, d=4)
1 2 3 4
>>> func(1, *(2,), c=3, **{'d':4})         # Same as func(1, 2, c=3, d=4)
1 2 3 4

This sort of code is convenient when you cannot predict the number of arguments that will be passed to a function when you write your script; you can build up a collection of arguments at runtime instead and call the function generically this way. Again, don’t confuse the */** starred-argument syntax in the function header and the function call—in the header it collects any number of arguments, while in the call it unpacks any number of arguments. In both, one star means positionals, and two applies to keywords.

The prior section’s examples may seem academic (if not downright esoteric), but they are used more often than you might expect. Some programs need to call arbitrary functions in a generic fashion, without knowing their names or arguments ahead of time. In fact, the real power of the special “varargs” call syntax is that you don’t need to know how many arguments a function call requires before you write a script. For example, you can use if logic to select from a set of functions and argument lists, and call any of them generically (functions in some of the following examples are hypothetical):

if sometest:
    action, args = func1, (1,)             # Call func1 with one arg in this case
else:
    action, args = func2, (1, 2, 3)        # Call func2 with three args here
...etc...
action(*args)                              # Dispatch generically

This leverages both the * form, and the fact that functions are objects that may be both referenced by, and called through, any variable. More generally, this varargs call syntax is useful anytime you cannot predict the arguments list. If your user selects an arbitrary function via a user interface, for instance, you may be unable to hardcode a function call when writing your script. To work around this, simply build up the arguments list with sequence operations, and call it with starred-argument syntax to unpack the arguments:

>>> ...define or import func3...
>>> args = (2,3)
>>> args += (4,)
>>> args
(2, 3, 4)
>>> func3(*args)

Because the arguments list is passed in as a tuple here, the program can build it at runtime. This technique also comes in handy for functions that test or time other functions. For instance, in the following code we support any function with any arguments by passing along whatever arguments were sent in (this is file tracer0.py in the book examples package):

def tracer(func, *pargs, **kargs):         # Accept arbitrary arguments
    print('calling:', func.__name__)
    return func(*pargs, **kargs)           # Pass along arbitrary arguments

def func(a, b, c, d):
    return a + b + c + d

print(tracer(func, 1, 2, c=3, d=4))

This code uses the built-in __name__ attribute attached to every function (as you might expect, it’s the function’s name string), and uses stars to collect and then unpack the arguments intended for the traced function. In other words, when this code is run, arguments are intercepted by the tracer and then propagated with varargs call syntax:

calling: func
10

For another example of this technique, see the preview near the end of the preceding chapter, where it was used to reset the built-in open function. We’ll code additional examples of such roles later in this book; see especially the sequence timing examples in Chapter 21 and the various decorator utilities we will code in Chapter 39. It’s a common technique in general tools.

Prior to Python 3.X, the effect of the *args and **args varargs call syntax could be achieved with a built-in function named apply. This original technique has been removed in 3.X because it is now redundant (3.X cleans up many such dusty tools that have been subsumed over the years). It’s still available in all Python 2.X releases, though, and you may come across it in older 2.X code.

In short, the following are equivalent prior to Python 3.X:

func(*pargs, **kargs)             # Newer call syntax: func(*sequence, **dict)
apply(func, pargs, kargs)         # Defunct built-in:  apply(func, sequence, dict)

For example, consider the following function, which accepts any number of positional or keyword arguments:

>>> def echo(*args, **kwargs): print(args, kwargs)

>>> echo(1, 2, a=3, b=4)
(1, 2) {'a': 3, 'b': 4}

In Python 2.X, we can call it generically with apply, or with the call syntax that is now required in 3.X:

>>> pargs = (1, 2)
>>> kargs = {'a':3, 'b':4}

>>> apply(echo, pargs, kargs)
(1, 2) {'a': 3, 'b': 4}

>>> echo(*pargs, **kargs)
(1, 2) {'a': 3, 'b': 4}

Both forms work for built-in functions in 2.X too (notice 2.X’s trailing L for its long integers):

>>> apply(pow, (2, 100))
1267650600228229401496703205376L
>>> pow(*(2, 100))
1267650600228229401496703205376L

The unpacking call syntax form is newer than the apply function, is preferred in general, and is required in 3.X. (Technically, it was added in 2.0, was documented as deprecated in 2.3, is still usable without warning in 2.7, and is gone in 3.0 and later.) Apart from its symmetry with the * collector forms in def headers, and the fact that it requires fewer keystrokes, the newer call syntax also allows us to pass along additional arguments without having to manually extend argument sequences or dictionaries:

>>> echo(0, c=5, *pargs, **kargs)      # Normal, keyword, *sequence, **dictionary
(0, 1, 2) {'a': 3, 'c': 5, 'b': 4}

That is, the call syntax form is more general. Since it’s required in 3.X, you should now disavow all knowledge of apply (unless, of course, it appears in 2.X code you must use or maintain...).

>>> def kwonly(a, *b, c):
        print(a, b, c)

>>> kwonly(1, 2, c=3)
1 (2,) 3
>>> kwonly(a=1, c=3)
1 () 3
>>> kwonly(1, 2, 3)
TypeError: kwonly() missing 1 required keyword-only argument: 'c'

>>> def kwonly(a, *, b, c):
        print(a, b, c)

>>> kwonly(1, c=3, b=2)
1 2 3
>>> kwonly(c=3, b=2, a=1)
1 2 3
>>> kwonly(1, 2, 3)
TypeError: kwonly() takes 1 positional argument but 3 were given
>>> kwonly(1)
TypeError: kwonly() missing 2 required keyword-only arguments: 'b' and 'c'

>>> def kwonly(a, *, b='spam', c='ham'):
        print(a, b, c)

>>> kwonly(1)
1 spam ham
>>> kwonly(1, c=3)
1 spam 3
>>> kwonly(a=1)
1 spam ham
>>> kwonly(c=3, b=2, a=1)
1 2 3
>>> kwonly(1, 2)
TypeError: kwonly() takes 1 positional argument but 2 were given

>>> def kwonly(a, *, b, c='spam'):
        print(a, b, c)

>>> kwonly(1, b='eggs')
1 eggs spam
>>> kwonly(1, c='eggs')
TypeError: kwonly() missing 1 required keyword-only argument: 'b'
>>> kwonly(1, 2)
TypeError: kwonly() takes 1 positional argument but 2 were given

>>> def kwonly(a, *, b=1, c, d=2):
        print(a, b, c, d)

>>> kwonly(3, c=4)
3 1 4 2
>>> kwonly(3, c=4, b=5)
3 5 4 2
>>> kwonly(3)
TypeError: kwonly() missing 1 required keyword-only argument: 'c'
>>> kwonly(1, 2, 3)
TypeError: kwonly() takes 1 positional argument but 3 were given

Finally, note that keyword-only arguments must be specified after a single star, not two—named arguments cannot appear after the **args arbitrary keywords form, and a ** can’t appear by itself in the arguments list. Both attempts generate a syntax error:

>>> def kwonly(a, **pargs, b, c):
SyntaxError: invalid syntax
>>> def kwonly(a, **, b, c):
SyntaxError: invalid syntax

This means that in a function header, keyword-only arguments must be coded before the **args arbitrary keywords form and after the *args arbitrary positional form, when both are present. Whenever an argument name appears before *args, it is a possibly default positional argument, not keyword-only:

>>> def f(a, *b, **d, c=6): print(a, b, c, d)          # Keyword-only before **!
SyntaxError: invalid syntax

>>> def f(a, *b, c=6, **d): print(a, b, c, d)          # Collect args in header

>>> f(1, 2, 3, x=4, y=5)                               # Default used
1 (2, 3) 6 {'y': 5, 'x': 4}

>>> f(1, 2, 3, x=4, y=5, c=7)                          # Override default
1 (2, 3) 7 {'y': 5, 'x': 4}

>>> f(1, 2, 3, c=7, x=4, y=5)                          # Anywhere in keywords
1 (2, 3) 7 {'y': 5, 'x': 4}

>>> def f(a, c=6, *b, **d): print(a, b, c, d)          # c is not keyword-only here!

>>> f(1, 2, 3, x=4)
1 (3,) 2 {'x': 4}

In fact, similar ordering rules hold true in function calls: when keyword-only arguments are passed, they must appear before a **args form. The keyword-only argument can be coded either before or after the *args, though, and may be included in **args:

>>> def f(a, *b, c=6, **d): print(a, b, c, d)          # KW-only between * and **

>>> f(1, *(2, 3), **dict(x=4, y=5))                    # Unpack args at call
1 (2, 3) 6 {'y': 5, 'x': 4}

>>> f(1, *(2, 3), **dict(x=4, y=5), c=7)               # Keywords before **args!
SyntaxError: invalid syntax

>>> f(1, *(2, 3), c=7, **dict(x=4, y=5))               # Override default
1 (2, 3) 7 {'y': 5, 'x': 4}

>>> f(1, c=7, *(2, 3), **dict(x=4, y=5))               # After or before *
1 (2, 3) 7 {'y': 5, 'x': 4}

>>> f(1, *(2, 3), **dict(x=4, y=5, c=7))               # Keyword-only in **
1 (2, 3) 7 {'y': 5, 'x': 4}

Trace through these cases on your own, in conjunction with the general argument-ordering rules described formally earlier. They may appear to be worst cases in the artificial examples here, but they can come up in real practice, especially for people who write libraries and tools for other Python programmers to use.

So why care about keyword-only arguments? In short, they make it easier to allow a function to accept both any number of positional arguments to be processed, and configuration options passed as keywords. While their use is optional, without keyword-only arguments extra work may be required to provide defaults for such options and to verify that no superfluous keywords were passed.

Imagine a function that processes a set of passed-in objects and allows a tracing flag to be passed:

process(X, Y, Z)                    # Use flag's default
process(X, Y, notify=True)          # Override flag default

Without keyword-only arguments we have to use both *args and **args and manually inspect the keywords, but with keyword-only arguments less code is required. The following guarantees that no positional argument will be incorrectly matched against notify and requires that it be a keyword if passed:

def process(*args, notify=False): ...

Since we’re going to see a more realistic example of this later in this chapter, in “Emulating the Python 3.X print Function,” I’ll postpone the rest of this story until then. For an additional example of keyword-only arguments in action, see the iteration options timing case study in Chapter 21. And for additional function definition enhancements in Python 3.X, stay tuned for the discussion of function annotation syntax in Chapter 19.