Now that we’ve seen how the iteration protocol works, let’s turn to one of its most common use cases. Together with for loops, list comprehensions are one of the most prominent contexts in which the iteration protocol is applied.
In the previous chapter, we learned how to use range to change a list as we step across it:
>>>L = [1, 2, 3, 4, 5]>>>for i in range(len(L)):...L[i] += 10... >>>L[11, 12, 13, 14, 15]
This works, but as I mentioned there, it may not be the optimal “best practice” approach in Python. Today, the list comprehension expression makes many such prior coding patterns obsolete. Here, for example, we can replace the loop with a single expression that produces the desired result list:
>>>L = [x + 10 for x in L]>>>L[21, 22, 23, 24, 25]
The net result is similar, but it requires less coding on our part and is likely to run substantially faster. The list comprehension isn’t exactly the same as the for loop statement version because it makes a new list object (which might matter if there are multiple references to the original list), but it’s close enough for most applications and is a common and convenient enough approach to merit a closer look here.
We met the list comprehension briefly in Chapter 4. Syntactically, its syntax is derived from a construct in set theory notation that applies an operation to each item in a set, but you don’t have to know set theory to use this tool. In Python, most people find that a list comprehension simply looks like a backward for loop.
To get a handle on the syntax, let’s dissect the prior section’s example in more detail:
L = [x + 10 for x in L]
List comprehensions are written in square brackets because they are ultimately a way to construct a new list. They begin with an arbitrary expression that we make up, which uses a loop variable that we make up (x + 10). That is followed by what you should now recognize as the header of a for loop, which names the loop variable, and an iterable object (for x in L).
To run the expression, Python executes an iteration across L inside the interpreter, assigning x to each item in turn, and collects the results of running the items through the expression on the left side. The result list we get back is exactly what the list comprehension says—a new list containing x + 10, for every x in L.
Technically speaking, list comprehensions are never really required because we can always build up a list of expression results manually with for loops that append results as we go:
>>>res = []>>>for x in L:...res.append(x + 10)... >>>res[31, 32, 33, 34, 35]
In fact, this is exactly what the list comprehension does internally.
However, list comprehensions are more concise to write, and because this code pattern of building up result lists is so common in Python work, they turn out to be very useful in many contexts. Moreover, depending on your Python and code, list comprehensions might run much faster than manual for loop statements (often roughly twice as fast) because their iterations are performed at C language speed inside the interpreter, rather than with manual Python code. Especially for larger data sets, there is often a major performance advantage to using this expression.
Let’s work through another common application of list comprehensions to explore them in more detail. Recall that the file object has a readlines method that loads the file into a list of line strings all at once:
>>>f = open('script2.py')>>>lines = f.readlines()>>>lines['import sys\n', 'print(sys.path)\n', 'x = 2\n', 'print(x ** 32)\n']
This works, but the lines in the result all include the newline character (\n) at the end. For many programs, the newline character gets in the way—we have to be careful to avoid double-spacing when printing, and so on. It would be nice if we could get rid of these newlines all at once, wouldn’t it?
Anytime we start thinking about performing an operation on each item in a sequence, we’re in the realm of list comprehensions. For example, assuming the variable lines is as it was in the prior interaction, the following code does the job by running each line in the list through the string rstrip method to remove whitespace on the right side (a line[:−1] slice would work, too, but only if we can be sure all lines are properly \n terminated, and this may not always be the case for the last line in a file):
>>>lines = [line.rstrip() for line in lines]>>>lines['import sys', 'print(sys.path)', 'x = 2', 'print(x ** 32)']
This works as planned. Because list comprehensions are an iteration context just like for loop statements, though, we don’t even have to open the file ahead of time. If we open it inside the expression, the list comprehension will automatically use the iteration protocol we met earlier in this chapter. That is, it will read one line from the file at a time by calling the file’s next handler method, run the line through the rstrip expression, and add it to the result list. Again, we get what we ask for—the rstrip result of a line, for every line in the file:
>>>lines = [line.rstrip() for line in open('script2.py')]>>>lines['import sys', 'print(sys.path)', 'x = 2', 'print(x ** 32)']
This expression does a lot implicitly, but we’re getting a lot of work for free here—Python scans the file by lines and builds a list of operation results automatically. It’s also an efficient way to code this operation: because most of this work is done inside the Python interpreter, it may be faster than an equivalent for statement, and won’t load a file into memory all at once like some other techniques. Again, especially for large files, the advantages of list comprehensions can be significant.
Besides their efficiency, list comprehensions are also remarkably expressive. In our example, we can run any string operation on a file’s lines as we iterate. To illustrate, here’s the list comprehension equivalent to the file iterator uppercase example we met earlier, along with a few other representative operations:
>>>[line.upper() for line in open('script2.py')]['IMPORT SYS\n', 'PRINT(SYS.PATH)\n', 'X = 2\n', 'PRINT(X ** 32)\n'] >>>[line.rstrip().upper() for line in open('script2.py')]['IMPORT SYS', 'PRINT(SYS.PATH)', 'X = 2', 'PRINT(X ** 32)'] >>>[line.split() for line in open('script2.py')][['import', 'sys'], ['print(sys.path)'], ['x', '=', '2'], ['print(x', '**', '32)']] >>>[line.replace(' ', '!') for line in open('script2.py')]['import!sys\n', 'print(sys.path)\n', 'x!=!2\n', 'print(x!**!32)\n'] >>>[('sys' in line, line[:5]) for line in open('script2.py')][(True, 'impor'), (True, 'print'), (False, 'x = 2'), (False, 'print')]
Recall that the method chaining in the second of these examples works because string methods return a new string, to which we can apply another string method. The last of these shows how we can also collect multiple results, as long as they’re wrapped in a collection like a tuple or list.
One fine point here: recall from Chapter 9 that file objects close themselves automatically when garbage-collected if still open. Hence, these list comprehensions will also automatically close the file when their temporary file object is garbage-collected after the expression runs. Outside CPython, though, you may want to code these to close manually if this is run in a loop, to ensure that file resources are freed immediately. See Chapter 9 for more on file close calls if you need a refresher on this.
In fact, list comprehensions can be even richer in practice, and even constitute a sort of iteration mini-language in their fullest forms. Let’s take a quick look at their syntax tools here.
As one particularly useful extension, the for loop nested in a comprehension expression can have an associated if clause to filter out of the result items for which the test is not true.
For example, suppose we want to repeat the prior section’s file-scanning example, but we need to collect only lines that begin with the letter p (perhaps the first character on each line is an action code of some sort). Adding an if filter clause to our expression does the trick:
>>>lines = [line.rstrip() for line in open('script2.py') if line[0] == 'p']>>>lines['print(sys.path)', 'print(x ** 32)']
Here, the if clause checks each line read from the file to see whether its first character is p; if not, the line is omitted from the result list. This is a fairly big expression, but it’s easy to understand if we translate it to its simple for loop statement equivalent. In general, we can always translate a list comprehension to a for statement by appending as we go and further indenting each successive part:
>>>res = []>>>for line in open('script2.py'):...if line[0] == 'p':...res.append(line.rstrip())... >>>res['print(sys.path)', 'print(x ** 32)']
This for statement equivalent works, but it takes up four lines instead of one and may run slower. In fact, you can squeeze a substantial amount of logic into a list comprehension when you need to—the following works like the prior but selects only lines that end in a digit (before the newline at the end), by filtering with a more sophisticated expression on the right side:
>>> [line.rstrip() for line in open('script2.py') if line.rstrip()[-1].isdigit()]
['x = 2']
As another if filter example, the first result in the following gives the total lines in a text file, and the second strips whitespace on both ends to omit blank links in the tally in just one line of code (this file, not included, contains lines describing typos found in the first draft of this book by my proofreader):
>>>fname = r'd:\books\5e\lp5e\draft1typos.txt'>>>len(open(fname).readlines())# All lines 263 >>>len([line for line in open(fname) if line.strip() != ''])# Nonblank lines 185
List comprehensions can become even more complex if we need them to—for instance, they may contain nested loops, coded as a series of for clauses. In fact, their full syntax allows for any number of for clauses, each of which can have an optional associated if clause.
For example, the following builds a list of the concatenation of x + y for every x in one string and every y in another. It effectively collects all the ordered combinations of the characters in two strings:
>>> [x + y for x in 'abc' for y in 'lmn']
['al', 'am', 'an', 'bl', 'bm', 'bn', 'cl', 'cm', 'cn']
Again, one way to understand this expression is to convert it to statement form by indenting its parts. The following is an equivalent, but likely slower, alternative way to achieve the same effect:
>>>res = []>>>for x in 'abc':...for y in 'lmn':...res.append(x + y)... >>>res['al', 'am', 'an', 'bl', 'bm', 'bn', 'cl', 'cm', 'cn']
Beyond this complexity level, though, list comprehension expressions can often become too compact for their own good. In general, they are intended for simple types of iterations; for more involved work, a simpler for statement structure will probably be easier to understand and modify in the future. As usual in programming, if something is difficult for you to understand, it’s probably not a good idea.
Because comprehensions are generally best taken in multiple doses, we’ll cut this story short here for now. We’ll revisit list comprehensions in Chapter 20 in the context of functional programming tools, and will define their syntax more formally and explore additional examples there. As we’ll find, comprehensions turn out to be just as related to functions as they are to looping statements.
A blanket qualification for all performance claims in this book, list comprehension or other: the relative speed of code depends much on the exact code tested and Python used, and is prone to change from release to release.
For example, in CPython 2.7 and 3.3 today, list comprehensions can still be twice as fast as corresponding for loops on some tests, but just marginally quicker on others, and perhaps even slightly slower on some when if filter clauses are used.
We’ll see how to time code in Chapter 21, and will learn how to interpret the file listcomp-speed.txt in the book examples package, which times this chapter’s code. For now, keep in mind that absolutes in performance benchmarks are as elusive as consensus in open source projects!