Later in the book, we’ll see that user-defined classes can implement the iteration protocol too. Because of this, it’s sometimes important to know which built-in tools make use of it—any tool that employs the iteration protocol will automatically work on any built-in type or user-defined class that provides it.
So far, I’ve been demonstrating iterators in the context of the for loop statement, because this part of the book is focused on statements. Keep in mind, though, that every built-in tool that scans from left to right across objects uses the iteration protocol. This includes the for loops we’ve seen:
>>>for line in open('script2.py'):# Use file iterators ...print(line.upper(), end='')... IMPORT SYS PRINT(SYS.PATH) X = 2 PRINT(X ** 32)
But also much more. For instance, list comprehensions and the map built-in function use the same protocol as their for loop cousin. When applied to a file, they both leverage the file object’s iterator automatically to scan line by line, fetching an iterator with __iter__ and calling __next__ each time through:
>>>uppers = [line.upper() for line in open('script2.py')]>>>uppers['IMPORT SYS\n', 'PRINT(SYS.PATH)\n', 'X = 2\n', 'PRINT(X ** 32)\n'] >>>map(str.upper, open('script2.py'))# map is itself an iterable in 3.X <map object at 0x00000000029476D8> >>>list(map(str.upper, open('script2.py')))['IMPORT SYS\n', 'PRINT(SYS.PATH)\n', 'X = 2\n', 'PRINT(X ** 32)\n']
We introduced the map call used here briefly in the preceding chapter (and in passing in Chapter 4); it’s a built-in that applies a function call to each item in the passed-in iterable object. map is similar to a list comprehension but is more limited because it requires a function instead of an arbitrary expression. It also returns an iterable object itself in Python 3.X, so we must wrap it in a list call to force it to give us all its values at once; more on this change later in this chapter. Because map, like the list comprehension, is related to both for loops and functions, we’ll also explore both again in Chapter 19 and Chapter 20.
Many of Python’s other built-ins process iterables, too. For example, sorted sorts items in an iterable; zip combines items from iterables; enumerate pairs items in an iterable with relative positions; filter selects items for which a function is true; and reduce runs pairs of items in an iterable through a function. All of these accept iterables, and zip, enumerate, and filter also return an iterable in Python 3.X, like map. Here they are in action running the file’s iterator automatically to read line by line:
>>>sorted(open('script2.py'))['import sys\n', 'print(sys.path)\n', 'print(x ** 32)\n', 'x = 2\n'] >>>list(zip(open('script2.py'), open('script2.py')))[('import sys\n', 'import sys\n'), ('print(sys.path)\n', 'print(sys.path)\n'), ('x = 2\n', 'x = 2\n'), ('print(x ** 32)\n', 'print(x ** 32)\n')] >>>list(enumerate(open('script2.py')))[(0, 'import sys\n'), (1, 'print(sys.path)\n'), (2, 'x = 2\n'), (3, 'print(x ** 32)\n')] >>>list(filter(bool, open('script2.py')))# nonempty=True ['import sys\n', 'print(sys.path)\n', 'x = 2\n', 'print(x ** 32)\n'] >>>import functools, operator>>>functools.reduce(operator.add, open('script2.py'))'import sys\nprint(sys.path)\nx = 2\nprint(x ** 32)\n'
All of these are iteration tools, but they have unique roles. We met zip and enumerate in the prior chapter; filter and reduce are in Chapter 19’s functional programming domain, so we’ll defer their details for now; the point to notice here is their use of the iteration protocol for files and other iterables.
We first saw the sorted function used here at work in, and we used it for dictionaries in Chapter 8. sorted is a built-in that employs the iteration protocol—it’s like the original list sort method, but it returns the new sorted list as a result and runs on any iterable object. Notice that, unlike map and others, sorted returns an actual list in Python 3.X instead of an iterable.
Interestingly, the iteration protocol is even more pervasive in Python today than the examples so far have demonstrated—essentially everything in Python’s built-in toolset that scans an object from left to right is defined to use the iteration protocol on the subject object. This even includes tools such as the list and tuple built-in functions (which build new objects from iterables), and the string join method (which makes a new string by putting a substring between strings contained in an iterable). Consequently, these will also work on an open file and automatically read one line at a time:
>>>list(open('script2.py'))['import sys\n', 'print(sys.path)\n', 'x = 2\n', 'print(x ** 32)\n'] >>>tuple(open('script2.py'))('import sys\n', 'print(sys.path)\n', 'x = 2\n', 'print(x ** 32)\n') >>>'&&'.join(open('script2.py'))'import sys\n&&print(sys.path)\n&&x = 2\n&&print(x ** 32)\n'
Even some tools you might not expect fall into this category. For example, sequence assignment, the in membership test, slice assignment, and the list’s extend method also leverage the iteration protocol to scan, and thus read a file by lines automatically:
>>>a, b, c, d = open('script2.py')# Sequence assignment >>>a, d('import sys\n', 'print(x ** 32)\n') >>>a, *b = open('script2.py')# 3.X extended form >>>a, b('import sys\n', ['print(sys.path)\n', 'x = 2\n', 'print(x ** 32)\n']) >>>'y = 2\n' in open('script2.py')# Membership test False >>>'x = 2\n' in open('script2.py')True >>>L = [11, 22, 33, 44]# Slice assignment >>>L[1:3] = open('script2.py')>>>L[11, 'import sys\n', 'print(sys.path)\n', 'x = 2\n', 'print(x ** 32)\n', 44] >>>L = [11]>>>L.extend(open('script2.py'))# list.extend method >>>L[11, 'import sys\n', 'print(sys.path)\n', 'x = 2\n', 'print(x ** 32)\n']
Per Chapter 8 extend iterates automatically, but append does not—use the latter (or similar) to add an iterable to a list without iterating, with the potential to be iterated across later:
>>>L = [11]>>>L.append(open('script2.py'))# list.append does not iterate >>>L[11, <_io.TextIOWrapper name='script2.py' mode='r' encoding='cp1252'>] >>>list(L[1])['import sys\n', 'print(sys.path)\n', 'x = 2\n', 'print(x ** 32)\n']
Iteration is a broadly supported and powerful model. Earlier, we saw that the built-in dict call accepts an iterable zip result, too (see Chapter 8 and Chapter 13). For that matter, so does the set call, as well as the newer set and dictionary comprehension expressions in Python 3.X and 2.7, which we met in Chapter 4, Chapter 5, and Chapter 8:
>>>set(open('script2.py')){'print(x ** 32)\n', 'import sys\n', 'print(sys.path)\n', 'x = 2\n'} >>>{line for line in open('script2.py')}{'print(x ** 32)\n', 'import sys\n', 'print(sys.path)\n', 'x = 2\n'} >>>{ix: line for ix, line in enumerate(open('script2.py'))}{0: 'import sys\n', 1: 'print(sys.path)\n', 2: 'x = 2\n', 3: 'print(x ** 32)\n'}
In fact, both set and dictionary comprehensions support the extended syntax of list comprehensions we met earlier in this chapter, including if tests:
>>>{line for line in open('script2.py') if line[0] == 'p'}{'print(x ** 32)\n', 'print(sys.path)\n'} >>>{ix: line for (ix, line) in enumerate(open('script2.py')) if line[0] == 'p'}{1: 'print(sys.path)\n', 3: 'print(x ** 32)\n'}
Like the list comprehension, both of these scan the file line by line and pick out lines that begin with the letter p. They also happen to build sets and dictionaries in the end, but we get a lot of work “for free” by combining file iteration and comprehension syntax. Later in the book we’ll meet a relative of comprehensions—generator expressions—that deploys the same syntax and works on iterables too, but is also iterable itself:
>>> list(line.upper() for line in open('script2.py')) # See Chapter 20
['IMPORT SYS\n', 'PRINT(SYS.PATH)\n', 'X = 2\n', 'PRINT(X ** 32)\n']
Other built-in functions support the iteration protocol as well, but frankly, some are harder to cast in interesting examples related to files! For example, the sum call computes the sum of all the numbers in any iterable; the any and all built-ins return True if any or all items in an iterable are True, respectively; and max and min return the largest and smallest item in an iterable, respectively. Like reduce, all of the tools in the following examples accept any iterable as an argument and use the iteration protocol to scan it, but return a single result:
>>>sum([3, 2, 4, 1, 5, 0])# sum expects numbers only 15 >>>any(['spam', '', 'ni'])True >>>all(['spam', '', 'ni'])False >>>max([3, 2, 5, 1, 4])5 >>>min([3, 2, 5, 1, 4])1
Strictly speaking, the max and min functions can be applied to files as well—they automatically use the iteration protocol to scan the file and pick out the lines with the highest and lowest string values, respectively (though I’ll leave valid use cases to your imagination):
>>>max(open('script2.py'))# Line with max/min string value 'x = 2\n' >>>min(open('script2.py'))'import sys\n'
There’s one last iteration context that’s worth mentioning, although it’s mostly a preview: in Chapter 18, we’ll learn that a special *arg form can be used in function calls to unpack a collection of values into individual arguments. As you can probably predict by now, this accepts any iterable, too, including files (see Chapter 18 for more details on this call syntax; Chapter 20 for a section that extends this idea to generator expressions; and Chapter 11 for tips on using the following’s 3.X print in 2.X as usual):
>>>def f(a, b, c, d): print(a, b, c, d, sep='&')... >>>f(1, 2, 3, 4)1&2&3&4 >>>f(*[1, 2, 3, 4])# Unpacks into arguments 1&2&3&4 >>> >>>f(*open('script2.py'))# Iterates by lines too! import sys &print(sys.path) &x = 2 &print(x ** 32)
In fact, because this argument-unpacking syntax in calls accepts iterables, it’s also possible to use the zip built-in to unzip zipped tuples, by making prior or nested zip results arguments for another zip call (warning: you probably shouldn’t read the following example if you plan to operate heavy machinery anytime soon!):
>>>X = (1, 2)>>>Y = (3, 4)>>> >>>list(zip(X, Y))# Zip tuples: returns an iterable [(1, 3), (2, 4)] >>> >>>A, B = zip(*zip(X, Y))# Unzip a zip! >>>A(1, 2) >>>B(3, 4)
Still other tools in Python, such as the range built-in and dictionary view objects, return iterables instead of processing them. To see how these have been absorbed into the iteration protocol in Python 3.X as well, we need to move on to the next section.