As mentioned early in this book, Python supports the procedural, object-oriented, and function programming paradigms. In fact, Python has a host of tools that most would considered functional in nature, which we enumerated in the preceding chapter—closures, generators, lambdas, comprehensions, maps, decorators, function objects, and more. These tools allow us to apply and combine functions in powerful ways, and often offer state retention and coding solutions that are alternatives to classes and OOP.
For instance, the prior chapter explored tools such as map and filter—key members of Python’s early functional programming toolset inspired by the Lisp language—that map operations over iterables and collect results. Because this is such a common task in Python coding, Python eventually sprouted a new expression—the list comprehension—that is even more flexible than the tools we just studied.
Per Python history, list comprehensions were originally inspired by a similar tool in the functional programming language Haskell, around the time of Python 2.0. In short, list comprehensions apply an arbitrary expression to items in an iterable, rather than applying a function. Accordingly, they can be more general tools. In later releases, the comprehension was extended to other roles—sets, dictionaries, and even the value generator expressions we’ll explore in this chapter. It’s not just for lists anymore.
We first met list comprehensions in Chapter 4’s preview, and studied them further in Chapter 14, in conjunction with looping statements. Because they’re also related to functional programming tools like the map and filter calls, though, we’ll resurrect the topic here for one last look. Technically, this feature is not tied to functions—as we’ll see, list comprehensions can be a more general tool than map and filter—but it is sometimes best understood by analogy to function-based alternatives.
Let’s work through an example that demonstrates the basics. As we saw in Chapter 7, Python’s built-in ord function returns the integer code point of a single character (the chr built-in is the converse—it returns the character for an integer code point). These happen to be ASCII codes if your characters fall into the ASCII character set’s 7-bit code point range:
>>> ord('s')
115
Now, suppose we wish to collect the ASCII codes of all characters in an entire string. Perhaps the most straightforward approach is to use a simple for loop and append the results to a list:
>>>res = []>>>for x in 'spam':res.append(ord(x))# Manual results collection >>>res[115, 112, 97, 109]
Now that we know about map, though, we can achieve similar results with a single function call without having to manage list construction in the code:
>>>res = list(map(ord, 'spam'))# Apply function to sequence (or other) >>>res[115, 112, 97, 109]
However, we can get the same results from a list comprehension expression—while map maps a function over an iterable, list comprehensions map an expression over a sequence or other iterable:
>>>res = [ord(x) for x in 'spam']# Apply expression to sequence (or other) >>>res[115, 112, 97, 109]
List comprehensions collect the results of applying an arbitrary expression to an iterable of values and return them in a new list. Syntactically, list comprehensions are enclosed in square brackets—to remind you that they construct lists. In their simple form, within the brackets you code an expression that names a variable followed by what looks like a for loop header that names the same variable. Python then collects the expression’s results for each iteration of the implied loop.
The effect of the preceding example is similar to that of the manual for loop and the map call. List comprehensions become more convenient, though, when we wish to apply an arbitrary expression to an iterable instead of a function:
>>> [x ** 2 for x in range(10)]
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
Here, we’ve collected the squares of the numbers 0 through 9 (we’re just letting the interactive prompt print the resulting list object; assign it to a variable if you need to retain it). To do similar work with a map call, we would probably need to invent a little function to implement the square operation. Because we won’t need this function elsewhere, we’d typically (but not necessarily) code it inline, with a lambda, instead of using a def statement elsewhere:
>>> list(map((lambda x: x ** 2), range(10)))
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
This does the same job, and it’s only a few keystrokes longer than the equivalent list comprehension. It’s also only marginally more complex (at least, once you understand the lambda). For more advanced kinds of expressions, though, list comprehensions will often require considerably less typing. The next section shows why.
List comprehensions are even more general than shown so far. For instance, as we learned in Chapter 14, you can code an if clause after the for to add selection logic. List comprehensions with if clauses can be thought of as analogous to the filter built-in discussed in the preceding chapter—they skip an iterable’s items for which the if clause is not true.
To demonstrate, following are both schemes picking up even numbers from 0 to 4; like the map list comprehension alternative of the prior section, the filter version here must invent a little lambda function for the test expression. For comparison, the equivalent for loop is shown here as well:
>>>[x for x in range(5) if x % 2 == 0][0, 2, 4] >>>list(filter((lambda x: x % 2 == 0), range(5)))[0, 2, 4] >>>res = []>>>for x in range(5):if x % 2 == 0:res.append(x)>>>res[0, 2, 4]
All of these use the modulus (remainder of division) operator, %, to detect even numbers: if there is no remainder after dividing a number by 2, it must be even. The filter call here is not much longer than the list comprehension either. However, we can combine an if clause and an arbitrary expression in our list comprehension, to give it the effect of a filter and a map, in a single expression:
>>> [x ** 2 for x in range(10) if x % 2 == 0]
[0, 4, 16, 36, 64]
This time, we collect the squares of the even numbers from 0 through 9: the for loop skips numbers for which the attached if clause on the right is false, and the expression on the left computes the squares. The equivalent map call would require a lot more work on our part—we would have to combine filter selections with map iteration, making for a noticeably more complex expression:
>>>list(map((lambda x: x**2), filter((lambda x: x % 2 == 0), range(10))) )[0, 4, 16, 36, 64]
In fact, list comprehensions are more general still. In their simplest form, you must always code an accumulation expression and a single for clause:
[expressionfortargetiniterable]
Though all other parts are optional, they allow richer iterations to be expressed—you can code any number of nested for loops in a list comprehension, and each may have an optional associated if test to act as a filter. The general structure of list comprehensions looks like this:
[expressionfortarget1initerable1ifcondition1fortarget2initerable2ifcondition2... fortargetNiniterableNifconditionN]
This same syntax is inherited by set and dictionary comprehensions as well as the generator expressions coming up, though these use different enclosing characters (curly braces or often-optional parentheses), and the dictionary comprehension begins with two expressions separated by a colon (for key and value).
We experimented with the if filter clause in the previous section. When for clauses are nested within a list comprehension, they work like equivalent nested for loop statements. For example:
>>>res = [x + y for x in [0, 1, 2] for y in [100, 200, 300]]>>>res[100, 200, 300, 101, 201, 301, 102, 202, 302]
This has the same effect as this substantially more verbose equivalent:
>>>res = []>>>for x in [0, 1, 2]:for y in [100, 200, 300]:res.append(x + y)>>>res[100, 200, 300, 101, 201, 301, 102, 202, 302]
Although list comprehensions construct list results, remember that they can iterate over any sequence or other iterable type. Here’s a similar bit of code that traverses strings instead of lists of numbers, and so collects concatenation results:
>>> [x + y for x in 'spam' for y in 'SPAM']
['sS', 'sP', 'sA', 'sM', 'pS', 'pP', 'pA', 'pM',
'aS', 'aP', 'aA', 'aM', 'mS', 'mP', 'mA', 'mM']
Each for clause can have an associated if filter, no matter how deeply the loops are nested—though use cases for the following sort of code, apart from perhaps multidimensional arrays, start to become more and more difficult to imagine at this level:
>>>[x + y for x in 'spam' if x in 'sm' for y in 'SPAM' if y in ('P', 'A')]['sP', 'sA', 'mP', 'mA'] >>>[x + y + z for x in 'spam' if x in 'sm'for y in 'SPAM' if y in ('P', 'A')for z in '123' if z > '1']['sP2', 'sP3', 'sA2', 'sA3', 'mP2', 'mP3', 'mA2', 'mA3']
Finally, here is a similar list comprehension that illustrates the effect of attached if selections on nested for clauses applied to numeric objects rather than strings:
>>> [(x, y) for x in range(5) if x % 2 == 0 for y in range(5) if y % 2 == 1]
[(0, 1), (0, 3), (2, 1), (2, 3), (4, 1), (4, 3)]
This expression combines even numbers from 0 through 4 with odd numbers from 0 through 4. The if clauses filter out items in each iteration. Here is the equivalent statement-based code:
>>>res = []>>>for x in range(5):if x % 2 == 0:for y in range(5):if y % 2 == 1:res.append((x, y))>>>res[(0, 1), (0, 3), (2, 1), (2, 3), (4, 1), (4, 3)]
Recall that if you’re confused about what a complex list comprehension does, you can always nest the list comprehension’s for and if clauses inside each other like this—indenting each clause successively further to the right—to derive the equivalent statements. The result is longer, but perhaps clearer in intent to some human readers on first glance, especially those more familiar with basic statements.
The map and filter equivalent of this last example would be wildly complex and deeply nested, so I won’t even try showing it here. I’ll leave its coding as an exercise for Zen masters, ex–Lisp programmers, and the criminally insane!
Not all list comprehensions are so artificial, of course. Let’s look at one more application to stretch a few synapses. As we saw in Chapter 4 and Chapter 8, one basic way to code matrixes (a.k.a. multidimensional arrays) in Python is with nested list structures. The following, for example, defines two 3 × 3 matrixes as lists of nested lists:
>>>M = [[1, 2, 3],[4, 5, 6],[7, 8, 9]]>>>N = [[2, 2, 2],[3, 3, 3],[4, 4, 4]]
Given this structure, we can always index rows, and columns within rows, using normal index operations:
>>>M[1]# Row 2 [4, 5, 6] >>>M[1][2]# Row 2, item 3 6
List comprehensions are powerful tools for processing such structures, though, because they automatically scan rows and columns for us. For instance, although this structure stores the matrix by rows, to collect the second column we can simply iterate across the rows and pull out the desired column, or iterate through positions in the rows and index as we go:
>>>[row[1] for row in M]# Column 2 [2, 5, 8] >>>[M[row][1] for row in (0, 1, 2)]# Using offsets [2, 5, 8]
Given positions, we can also easily perform tasks such as pulling out a diagonal. The first of the following expressions uses range to generate the list of offsets and then indexes with the row and column the same, picking out M[0][0], then M[1][1], and so on. The second scales the column index to fetch M[0][2], M[1][1], etc. (we assume the matrix has the same number of rows and columns):
>>>[M[i][i] for i in range(len(M))]# Diagonals [1, 5, 9] >>>[M[i][len(M)-1-i] for i in range(len(M))][3, 5, 7]
Changing such a matrix in place requires assignment to offsets (use range twice if shapes differ):
>>>L = [[1, 2, 3], [4, 5, 6]]>>>for i in range(len(L)):for j in range(len(L[i])):# Update in placeL[i][j] += 10>>>L[[11, 12, 13], [14, 15, 16]]
We can’t really do the same with list comprehensions, as they make new lists, but we could always assign their results to the original name for a similar effect. For example, we can apply an operation to every item in a matrix, producing results in either a simple vector or a matrix of the same shape:
>>>[col + 10 for row in M for col in row]# Assign to M to retain new value [11, 12, 13, 14, 15, 16, 17, 18, 19] >>>[[col + 10 for col in row] for row in M][[11, 12, 13], [14, 15, 16], [17, 18, 19]]
To understand these, translate to their simple statement form equivalents that follow—indent parts that are further to the right in the expression (as in the first loop in the following), and make a new list when comprehensions are nested on the left (like the second loop in the following). As its statement equivalent makes clearer, the second expression in the preceding works because the row iteration is an outer loop: for each row, it runs the nested column iteration to build up one row of the result matrix:
>>>res = []>>>for row in M:# Statement equivalentsfor col in row: # Indent parts further rightres.append(col + 10)>>>res[11, 12, 13, 14, 15, 16, 17, 18, 19] >>>res = []>>>for row in M:tmp = []# Left-nesting starts new listfor col in row:tmp.append(col + 10)res.append(tmp)>>>res[[11, 12, 13], [14, 15, 16], [17, 18, 19]]
Finally, with a bit of creativity, we can also use list comprehensions to combine values of multiple matrixes. The following first builds a flat list that contains the result of multiplying the matrixes pairwise, and then builds a nested list structure having the same values by nesting list comprehensions again:
>>>M[[1, 2, 3], [4, 5, 6], [7, 8, 9]] >>>N[[2, 2, 2], [3, 3, 3], [4, 4, 4]] >>>[M[row][col] * N[row][col] for row in range(3) for col in range(3)][2, 4, 6, 12, 15, 18, 28, 32, 36] >>>[[M[row][col] * N[row][col] for col in range(3)] for row in range(3)][[2, 4, 6], [12, 15, 18], [28, 32, 36]]
This last expression works because the row iteration is an outer loop again; it’s equivalent to this statement-based code:
res = []
for row in range(3):
tmp = []
for col in range(3):
tmp.append(M[row][col] * N[row][col])
res.append(tmp)
And for more fun, we can use zip to pair items to be multiplied—the following comprehension and loop statement forms both produce the same list-of-lists pairwise multiplication result as the last preceding example (and because zip is a generator of values in 3.X, this isn’t as inefficient as it may seem):
[[col1 * col2 for (col1, col2) in zip(row1, row2)] for (row1, row2) in zip(M, N)]
res = []
for (row1, row2) in zip(M, N):
tmp = []
for (col1, col2) in zip(row1, row2):
tmp.append(col1 * col2)
res.append(tmp)
Compared to their statement equivalents, the list comprehension versions here require only one line of code, might run substantially faster for large matrixes, and just might make your head explode! Which brings us to the next section.
With such generality, list comprehensions can quickly become, well, incomprehensible, especially when nested. Some programming tasks are inherently complex, and we can’t sugarcoat them to make them any simpler than they are (see the upcoming permutations for a prime example). Tools like comprehensions are powerful solutions when used wisely, and there’s nothing inherently wrong with using them in your scripts.
At the same time, code like that of the prior section may push the complexity envelope more than it should—and, frankly, tends to disproportionately pique the interest of those holding the darker and misguided assumption that code obfuscation somehow implies talent. Because such tools tend to appeal to some people more than they probably should, I need to be clear about their scope here.
This book demonstrates advanced comprehensions to teach, but in the real world, using complicated and tricky code where not warranted is both bad engineering and bad software citizenship. To repurpose a line from the first chapter: programming is not about being clever and obscure—it’s about how clearly your program communicates its purpose.
Or, to quote from Python’s import this motto:
Simple is better than complex.
Writing complicated comprehension code may be a fun academic recreation, but it doesn’t have a place in programs that others will someday need to understand.
Consequently, my advice is to use simple for loops when getting started with Python, and comprehensions or map in isolated cases where they are easy to apply. The “keep it simple” rule applies here as always: code conciseness is a much less important goal than code readability. If you have to translate code to statements to understand it, it should probably be statements in the first place. In other words, the age-old acronym KISS still applies: Keep It Simple—followed either by a word that is today too sexist (Sir), or another that is too colorful for a family-oriented book like this...
However, in this case, there is currently a substantial performance advantage to the extra complexity: based on tests run under Python today, map calls can be twice as fast as equivalent for loops, and list comprehensions are often faster than map calls. This speed difference can vary per usage pattern and Python, but is generally due to the fact that map and list comprehensions run at C language speed inside the interpreter, which is often much faster than stepping through Python for loop bytecode within the PVM.
In addition, list comprehensions offer a code conciseness that’s compelling and even warranted when that reduction in size doesn’t also imply a reduction in meaning for the next programmer. Moreover, many find the expressiveness of comprehensions to be a powerful ally. Because map and list comprehensions are both expressions, they also can show up syntactically in places that for loop statements cannot, such as in the bodies of lambda functions, within list and dictionary literals, and more.
Because of this, list comprehensions and map calls are worth knowing and using for simpler kinds of iterations, especially if your application’s speed is an important consideration. Still, because for loops make logic more explicit, they are generally recommended on the grounds of simplicity, and often make for more straightforward code. When used, you should try to keep your map calls and list comprehensions simple; for more complex tasks, use full statements instead.
As I’ve stated before, performance generalizations like those just given here can depend on call patterns, as well as changes and optimizations in Python itself. Recent Python releases have sped up the simple for loop statement, for example. On some code, though, list comprehensions are still substantially faster than for loops and even faster than map, though map can still win when the alternatives must apply a function call, built-in functions or otherwise. At least until this story changes arbitrarily—to time these alternatives yourself, see tools in the standard library’s time module or in the newer timeit module added in Release 2.4, or stay tuned for the extended coverage of both of these in the next chapter, where we’ll prove the prior paragraph’s claims.