Functional Programming Tools

One of the more common things programs do with lists and other sequences is apply an operation to each item and collect the results—selecting columns in database tables, incrementing pay fields of employees in a company, parsing email attachments, and so on. Python has multiple tools that make such collection-wide operations easy to code. For instance, updating all the counters in a list can be done easily with a for loop:

>>> counters = [1, 2, 3, 4]
>>>
>>> updated = []
>>> for x in counters:
        updated.append(x + 10)                 # Add 10 to each item

>>> updated
[11, 12, 13, 14]

But because this is such a common operation, Python also provides built-ins that do most of the work for you. The map function applies a passed-in function to each item in an iterable object and returns a list containing all the function call results. For example:

>>> def inc(x): return x + 10                  # Function to be run

>>> list(map(inc, counters))                   # Collect results
[11, 12, 13, 14]

We met map briefly in Chapter 13 and Chapter 14, as a way to apply a built-in function to items in an iterable. Here, we make more general use of it by passing in a user-defined function to be applied to each item in the list—map calls inc on each list item and collects all the return values into a new list. Remember that map is an iterable in Python 3.X, so a list call is used to force it to produce all its results for display here; this isn’t necessary in 2.X (see Chapter 14 if you’ve forgotten this requirement).

Because map expects a function to be passed in and applied, it also happens to be one of the places where lambda commonly appears:

>>> list(map((lambda x: x + 3), counters))     # Function expression
[4, 5, 6, 7]

Here, the function adds 3 to each item in the counters list; as this little function isn’t needed elsewhere, it was written inline as a lambda. Because such uses of map are equivalent to for loops, with a little extra code you can always code a general mapping utility yourself:

>>> def mymap(func, seq):
        res = []
        for x in seq: res.append(func(x))
        return res

Assuming the function inc is still as it was when it was shown previously, we can map it across a sequence (or other iterable) with either the built-in or our equivalent:

>>> list(map(inc, [1, 2, 3]))             # Built-in is an iterable
[11, 12, 13]
>>> mymap(inc, [1, 2, 3])                 # Ours builds a list (see generators)
[11, 12, 13]

However, as map is a built-in, it’s always available, always works the same way, and has some performance benefits (as we’ll prove in Chapter 21, it’s faster than a manually coded for loop in some usage modes). Moreover, map can be used in more advanced ways than shown here. For instance, given multiple sequence arguments, it sends items taken from sequences in parallel as distinct arguments to the function:

>>> pow(3, 4)                             # 3**4
81
>>> list(map(pow, [1, 2, 3], [2, 3, 4]))  # 1**2, 2**3, 3**4
[1, 8, 81]

With multiple sequences, map expects an N-argument function for N sequences. Here, the pow function takes two arguments on each call—one from each sequence passed to map. It’s not much extra work to simulate this multiple-sequence generality in code, too, but we’ll postpone doing so until later in the next chapter, after we’ve met some additional iteration tools.

The map call is similar to the list comprehension expressions we studied in Chapter 14 and will revisit in the next chapter from a functional perspective:

>>> list(map(inc, [1, 2, 3, 4]))
[11, 12, 13, 14]
>>> [inc(x) for x in [1, 2, 3, 4]]        # Use () parens to generate items instead
[11, 12, 13, 14]

In some cases, map may be faster to run than a list comprehension (e.g., when mapping a built-in function), and it may also require less coding. On the other hand, because map applies a function call to each item instead of an arbitrary expression, it is a somewhat less general tool, and often requires extra helper functions or lambdas. Moreover, wrapping a comprehension in parentheses instead of square brackets creates an object that generates values on request to save memory and increase responsiveness, much like map in 3.X—a topic we’ll take up in the next chapter.

The map function is a primary and relatively straightforward representative of Python’s functional programming toolset. Its close relatives, filter and reduce, select an iterable’s items based on a test function and apply functions to item pairs, respectively.

Because it also returns an iterable, filter (like range) requires a list call to display all its results in 3.X. For example, the following filter call picks out items in a sequence that are greater than zero:

>>> list(range(−5, 5))                                   # An iterable in 3.X
[−5, −4, −3, −2, −1, 0, 1, 2, 3, 4]

>>> list(filter((lambda x: x > 0), range(−5, 5)))        # An iterable in 3.X
[1, 2, 3, 4]

We met filter briefly earlier in a Chapter 12 sidebar, and while exploring 3.X iterables in Chapter 14. Items in the sequence or iterable for which the function returns a true result are added to the result list. Like map, this function is roughly equivalent to a for loop, but it is built-in, concise, and often fast:

>>> res = []
>>> for x in range(−5, 5):                               # The statement equivalent
        if x > 0:
            res.append(x)

>>> res
[1, 2, 3, 4]

Also like map, filter can be emulated by list comprehension syntax with often-simpler results (especially when it can avoid creating a new function), and with a similar generator expression when delayed production of results is desired—though we’ll save the rest of this story for the next chapter:

>>> [x for x in range(−5, 5) if x > 0]                   # Use () to generate items
[1, 2, 3, 4]

The functional reduce call, which is a simple built-in function in 2.X but lives in the functools module in 3.X, is more complex. It accepts an iterable to process, but it’s not an iterable itself—it returns a single result. Here are two reduce calls that compute the sum and product of the items in a list:

>>> from functools import reduce                         # Import in 3.X, not in 2.X
>>> reduce((lambda x, y: x + y), [1, 2, 3, 4])
10
>>> reduce((lambda x, y: x * y), [1, 2, 3, 4])
24

At each step, reduce passes the current sum or product, along with the next item from the list, to the passed-in lambda function. By default, the first item in the sequence initializes the starting value. To illustrate, here’s the for loop equivalent to the first of these calls, with the addition hardcoded inside the loop:

>>> L = [1,2,3,4]
>>> res = L[0]
>>> for x in L[1:]:
        res = res + x

>>> res
10

Coding your own version of reduce is actually fairly straightforward. The following function emulates most of the built-in’s behavior and helps demystify its operation in general:

>>> def myreduce(function, sequence):
        tally = sequence[0]
        for next in sequence[1:]:
            tally = function(tally, next)
        return tally

>>> myreduce((lambda x, y: x + y), [1, 2, 3, 4, 5])
15
>>> myreduce((lambda x, y: x * y), [1, 2, 3, 4, 5])
120

The built-in reduce also allows an optional third argument placed before the items in the sequence to serve as a default result when the sequence is empty, but we’ll leave this extension as a suggested exercise.

If this coding technique has sparked your interest, you might also be interested in the standard library operator module, which provides functions that correspond to built-in expressions and so comes in handy for some uses of functional tools (see Python’s library manual for more details on this module):

>>> import operator, functools
>>> functools.reduce(operator.add, [2, 4, 6])        # Function-based +
12
>>> functools.reduce((lambda x, y: x + y), [2, 4, 6])
12

Together, map, filter, and reduce support powerful functional programming techniques. As mentioned, many observers would also extend the functional programming toolset in Python to include nested function scope closures (a.k.a. factory functions) and the anonymous function lambda—both discussed earlier—as well as generators and comprehensions, topics we will return to in the next chapter.