The for loop we just studied subsumes most counter-style loops. It’s generally simpler to code and often quicker to run than a while, so it’s the first tool you should reach for whenever you need to step through a sequence or other iterable. In fact, as a general rule, you should resist the temptation to count things in Python—its iteration tools automate much of the work you do to loop over collections in lower-level languages like C.
Still, there are situations where you will need to iterate in more specialized ways. For example, what if you need to visit every second or third item in a list, or change the list along the way? How about traversing more than one sequence in parallel, in the same for loop? What if you need indexes too?
You can always code such unique iterations with a while loop and manual indexing, but Python provides a set of built-ins that allow you to specialize the iteration in a for:
The built-in range function (available since Python 0.X) produces a series of successively higher integers, which can be used as indexes in a for.
The built-in zip function (available since Python 2.0) returns a series of parallel-item tuples, which can be used to traverse multiple sequences in a for.
The built-in enumerate function (available since Python 2.3) generates both the values and indexes of items in an iterable, so we don’t need to count manually.
The built-in map function (available since Python 1.0) can have a similar effect to zip in Python 2.X, though this role is removed in 3.X.
Because for loops may run quicker than while-based counter loops, though, it’s to your advantage to use tools like these that allow you to use for whenever possible. Let’s look at each of these built-ins in turn, in the context of common use cases. As we’ll see, their usage may differ slightly between 2.X and 3.X, and some of their applications are more valid than others.
Our first loop-related function, range, is really a general tool that can be used in a variety of contexts. We met it briefly in Chapter 4. Although it’s used most often to generate indexes in a for, you can use it anywhere you need a series of integers. In Python 2.X range creates a physical list; in 3.X, range is an iterable that generates items on demand, so we need to wrap it in a list call to display its results all at once in 3.X only:
>>> list(range(5)), list(range(2, 5)), list(range(0, 10, 2))
([0, 1, 2, 3, 4], [2, 3, 4], [0, 2, 4, 6, 8])
With one argument, range generates a list of integers from zero up to but not including the argument’s value. If you pass in two arguments, the first is taken as the lower bound. An optional third argument can give a step; if it is used, Python adds the step to each successive integer in the result (the step defaults to +1). Ranges can also be nonpositive and nonascending, if you want them to be:
>>>list(range(−5, 5))[−5, −4, −3, −2, −1, 0, 1, 2, 3, 4] >>>list(range(5, −5, −1))[5, 4, 3, 2, 1, 0, −1, −2, −3, −4]
We’ll get more formal about iterables like this one in Chapter 14. There, we’ll also see that Python 2.X has a cousin named xrange, which is like its range but doesn’t build the result list in memory all at once. This is a space optimization, which is subsumed in 3.X by the generator behavior of its range.
Although such range results may be useful all by themselves, they tend to come in most handy within for loops. For one thing, they provide a simple way to repeat an action a specific number of times. To print three lines, for example, use a range to generate the appropriate number of integers:
>>>for i in range(3):...print(i, 'Pythons')... 0 Pythons 1 Pythons 2 Pythons
Note that for loops force results from range automatically in 3.X, so we don’t need to use a list wrapper here in 3.X (in 2.X we get a temporary list unless we call xrange instead).
The range call is also sometimes used to iterate over a sequence indirectly, though it’s often not the best approach in this role. The easiest and generally fastest way to step through a sequence exhaustively is always with a simple for, as Python handles most of the details for you:
>>>X = 'spam'>>>for item in X: print(item, end=' ')# Simple iteration ... s p a m
Internally, the for loop handles the details of the iteration automatically when used this way. If you really need to take over the indexing logic explicitly, you can do it with a while loop:
>>>i = 0>>>while i < len(X):# while loop iteration ...print(X[i], end=' ')...i += 1... s p a m
You can also do manual indexing with a for, though, if you use range to generate a list of indexes to iterate through. It’s a multistep process, but it’s sufficient to generate offsets, rather than the items at those offsets:
>>>X'spam' >>>len(X)# Length of string 4 >>>list(range(len(X)))# All legal offsets into X [0, 1, 2, 3] >>> >>>for i in range(len(X)): print(X[i], end=' ')# Manual range/len iteration ... s p a m
Note that because this example is stepping over a list of offsets into X, not the actual items of X, we need to index back into X within the loop to fetch each item. If this seems like overkill, though, it’s because it is: there’s really no reason to work this hard in this example.
Although the range/len combination suffices in this role, it’s probably not the best option. It may run slower, and it’s also more work than we need to do. Unless you have a special indexing requirement, you’re better off using the simple for loop form in Python:
>>> for item in X: print(item, end=' ') # Use simple iteration if you can
As a general rule, use for instead of while whenever possible, and don’t use range calls in for loops except as a last resort. This simpler solution is almost always better. Like every good rule, though, there are plenty of exceptions—as the next section demonstrates.
Though not ideal for simple sequence scans, the coding pattern used in the prior example does allow us to do more specialized sorts of traversals when required. For example, some algorithms can make use of sequence reordering—to generate alternatives in searches, to test the effect of different value orderings, and so on. Such cases may require offsets in order to pull sequences apart and put them back together, as in the following; the range’s integers provide a repeat count in the first, and a position for slicing in the second:
>>>S = 'spam'>>>for i in range(len(S)):# For repeat counts 0..3 ...S = S[1:] + S[:1]# Move front item to end ...print(S, end=' ')... pams amsp mspa spam >>>S'spam' >>>for i in range(len(S)):# For positions 0..3 ...X = S[i:] + S[:i]# Rear part + front part ...print(X, end=' ')... spam pams amsp mspa
Trace through these one iteration at a time if they seem confusing. The second creates the same results as the first, though in a different order, and doesn’t change the original variable as it goes. Because both slice to obtain parts to concatenate, they also work on any type of sequence, and return sequences of the same type as that being shuffled—if you shuffle a list, you create reordered lists:
>>>L = [1, 2, 3]>>>for i in range(len(L)):...X = L[i:] + L[:i]# Works on any sequence type ...print(X, end=' ')... [1, 2, 3] [2, 3, 1] [3, 1, 2]
We’ll make use of code like this to test functions with different argument orderings in Chapter 18, and will extend it to functions, generators, and more complete permutations in Chapter 20—it’s a widely useful tool.
Cases like that of the prior section are valid applications for the range/len combination. We might also use this technique to skip items as we go:
>>>S = 'abcdefghijk'>>>list(range(0, len(S), 2))[0, 2, 4, 6, 8, 10] >>>for i in range(0, len(S), 2): print(S[i], end=' ')... a c e g i k
Here, we visit every second item in the string S by stepping over the generated range list. To visit every third item, change the third range argument to be 3, and so on. In effect, using range this way lets you skip items in loops while still retaining the simplicity of the for loop construct.
In most cases, though, this is also probably not the “best practice” technique in Python today. If you really mean to skip items in a sequence, the extended three-limit form of the slice expression, presented in Chapter 7, provides a simpler route to the same goal. To visit every second character in S, for example, slice with a stride of 2:
>>>S = 'abcdefghijk'>>>for c in S[::2]:print(c, end=' ')... a c e g i k
The result is the same, but substantially easier for you to write and for others to read. The potential advantage to using range here instead is space: slicing makes a copy of the string in both 2.X and 3.X, while range in 3.X and xrange in 2.X do not create a list; for very large strings, they may save memory.
Another common place where you may use the range/len combination with for is in loops that change a list as it is being traversed. Suppose, for example, that you need to add 1 to every item in a list (maybe you’re giving everyone a raise in an employee database list). You can try this with a simple for loop, but the result probably won’t be exactly what you want:
>>>L = [1, 2, 3, 4, 5]>>>for x in L:...x += 1# Changes x, not L ... >>>L[1, 2, 3, 4, 5] >>>x6
This doesn’t quite work—it changes the loop variable x, not the list L. The reason is somewhat subtle. Each time through the loop, x refers to the next integer already pulled out of the list. In the first iteration, for example, x is integer 1. In the next iteration, the loop body sets x to a different object, integer 2, but it does not update the list where 1 originally came from; it’s a piece of memory separate from the list.
To really change the list as we march across it, we need to use indexes so we can assign an updated value to each position as we go. The range/len combination can produce the required indexes for us:
>>>L = [1, 2, 3, 4, 5]>>>for i in range(len(L)):# Add one to each item in L ...L[i] += 1# Or L[i] = L[i] + 1 ... >>>L[2, 3, 4, 5, 6]
When coded this way, the list is changed as we proceed through the loop. There is no way to do the same with a simple for x in L:–style loop, because such a loop iterates through actual items, not list positions. But what about the equivalent while loop? Such a loop requires a bit more work on our part, and might run more slowly depending on your Python (it does on 2.7 and 3.3, though less so on 3.3—we’ll see how to verify this in Chapter 21):
>>>i = 0>>>while i < len(L):...L[i] += 1...i += 1... >>>L[3, 4, 5, 6, 7]
Here again, though, the range solution may not be ideal either. A list comprehension expression of the form:
[x + 1 for x in L]
likely runs faster today and would do similar work, albeit without changing the original list in place (we could assign the expression’s new list object result back to L, but this would not update any other references to the original list). Because this is such a central looping concept, we’ll save a complete exploration of list comprehensions for the next chapter, and continue this story there.
Our next loop coding technique extends a loop’s scope. As we’ve seen, the range built-in allows us to traverse sequences with for in a nonexhaustive fashion. In the same spirit, the built-in zip function allows us to use for loops to visit multiple sequences in parallel—not overlapping in time, but during the same loop. In basic operation, zip takes one or more sequences as arguments and returns a series of tuples that pair up parallel items taken from those sequences. For example, suppose we’re working with two lists (a list of names and addresses paired by position, perhaps):
>>>L1 = [1,2,3,4]>>>L2 = [5,6,7,8]
To combine the items in these lists, we can use zip to create a list of tuple pairs. Like range, zip is a list in Python 2.X, but an iterable object in 3.X where we must wrap it in a list call to display all its results at once (again, there’s more on iterables coming up in the next chapter):
>>>zip(L1, L2)<zip object at 0x026523C8> >>>list(zip(L1, L2))# list() required in 3.X, not 2.X [(1, 5), (2, 6), (3, 7), (4, 8)]
Such a result may be useful in other contexts as well, but when wedded with the for loop, it supports parallel iterations:
>>>for (x, y) in zip(L1, L2):...print(x, y, '--', x+y)... 1 5 -- 6 2 6 -- 8 3 7 -- 10 4 8 -- 12
Here, we step over the result of the zip call—that is, the pairs of items pulled from the two lists. Notice that this for loop again uses the tuple assignment form we met earlier to unpack each tuple in the zip result. The first time through, it’s as though we ran the assignment statement (x, y) = (1, 5).
The net effect is that we scan both L1 and L2 in our loop. We could achieve a similar effect with a while loop that handles indexing manually, but it would require more typing and would likely run more slowly than the for/zip approach.
Strictly speaking, the zip function is more general than this example suggests. For instance, it accepts any type of sequence (really, any iterable object, including files), and it accepts more than two arguments. With three arguments, as in the following example, it builds a list of three-item tuples with items from each sequence, essentially projecting by columns (technically, we get an N-ary tuple for N arguments):
>>>T1, T2, T3 = (1,2,3), (4,5,6), (7,8,9)>>>T3(7, 8, 9) >>>list(zip(T1, T2, T3))# Three tuples for three arguments [(1, 4, 7), (2, 5, 8), (3, 6, 9)]
Moreover, zip truncates result tuples at the length of the shortest sequence when the argument lengths differ. In the following, we zip together two strings to pick out characters in parallel, but the result has only as many tuples as the length of the shortest sequence:
>>>S1 = 'abc'>>>S2 = 'xyz123'>>> >>>list(zip(S1, S2))# Truncates at len(shortest) [('a', 'x'), ('b', 'y'), ('c', 'z')]
In Python 2.X only, the related built-in map function pairs items from sequences in a similar fashion when passed None for its function argument, but it pads shorter sequences with None if the argument lengths differ instead of truncating to the shortest length:
>>>S1 = 'abc'>>>S2 = 'xyz123'>>>map(None, S1, S2)# 2.X only: pads to len(longest) [('a', 'x'), ('b', 'y'), ('c', 'z'), (None, '1'), (None, '2'), (None,'3')]
This example is using a degenerate form of the map built-in, which is no longer supported in 3.X. Normally, map takes a function and one or more sequence arguments and collects the results of calling the function with parallel items taken from the sequence(s).
We’ll study map in detail in Chapter 19 and Chapter 20, but as a brief example, the following maps the built-in ord function across each item in a string and collects the results (like zip, map is a value generator in 3.X and so must be passed to list to collect all its results at once in 3.X only):
>>> list(map(ord, 'spam'))
[115, 112, 97, 109]
This works the same as the following loop statement, but map is often quicker, as Chapter 21 will show:
>>>res = []>>>for c in 'spam': res.append(ord(c))>>>res[115, 112, 97, 109]
Version skew note: The degenerate form of map using a function argument of None is no longer supported in Python 3.X, because it largely overlaps with zip (and was, frankly, a bit at odds with map’s function-application purpose). In 3.X, either use zip or write loop code to pad results yourself. In fact, we’ll see how to write such loop code in Chapter 20, after we’ve had a chance to study some additional iteration concepts.
Let’s look at another zip use case. Chapter 8 suggested that the zip call used here can also be handy for generating dictionaries when the sets of keys and values must be computed at runtime. Now that we’re becoming proficient with zip, let’s explore more fully how it relates to dictionary construction. As you’ve learned, you can always create a dictionary by coding a dictionary literal, or by assigning to keys over time:
>>>D1 = {'spam':1, 'eggs':3, 'toast':5}>>>D1{'eggs': 3, 'toast': 5, 'spam': 1} >>>D1 = {}>>>D1['spam'] = 1>>>D1['eggs'] = 3>>>D1['toast'] = 5
What to do, though, if your program obtains dictionary keys and values in lists at runtime, after you’ve coded your script? For example, say you had the following keys and values lists, collected from a user, parsed from a file, or obtained from another dynamic source:
>>>keys = ['spam', 'eggs', 'toast']>>>vals = [1, 3, 5]
One solution for turning those lists into a dictionary would be to zip the lists and step through them in parallel with a for loop:
>>>list(zip(keys, vals))[('spam', 1), ('eggs', 3), ('toast', 5)] >>>D2 = {}>>>for (k, v) in zip(keys, vals): D2[k] = v... >>>D2{'eggs': 3, 'toast': 5, 'spam': 1}
It turns out, though, that in Python 2.2 and later you can skip the for loop altogether and simply pass the zipped keys/values lists to the built-in dict constructor call:
>>>keys = ['spam', 'eggs', 'toast']>>>vals = [1, 3, 5]>>>D3 = dict(zip(keys, vals))>>>D3{'eggs': 3, 'toast': 5, 'spam': 1}
The built-in name dict is really a type name in Python (you’ll learn more about type names, and subclassing them, in Chapter 32). Calling it achieves something like a list-to-dictionary conversion, but it’s really an object construction request.
In the next chapter we’ll explore the related but richer concept, the list comprehension, which builds lists in a single expression; we’ll also revisit Python 3.X and 2.7 dictionary comprehensions, an alternative to the dict call for zipped key/value pairs:
>>> {k: v for (k, v) in zip(keys, vals)}
{'eggs': 3, 'toast': 5, 'spam': 1}
Our final loop helper function is designed to support dual usage modes. Earlier, we discussed using range to generate the offsets of items in a string, rather than the items at those offsets. In some programs, though, we need both: the item to use, plus an offset as we go. Traditionally, this was coded with a simple for loop that also kept a counter of the current offset:
>>>S = 'spam'>>>offset = 0>>>for item in S:...print(item, 'appears at offset', offset)...offset += 1... s appears at offset 0 p appears at offset 1 a appears at offset 2 m appears at offset 3
This works, but in all recent Python 2.X and 3.X releases (since 2.3) a new built-in named enumerate does the job for us—its net effect is to give loops a counter “for free,” without sacrificing the simplicity of automatic iteration:
>>>S = 'spam'>>>for (offset, item) in enumerate(S):...print(item, 'appears at offset', offset)... s appears at offset 0 p appears at offset 1 a appears at offset 2 m appears at offset 3
The enumerate function returns a generator object—a kind of object that supports the iteration protocol that we will study in the next chapter and will discuss in more detail in the next part of the book. In short, it has a method called by the next built-in function, which returns an (index, value) tuple each time through the loop. The for steps through these tuples automatically, which allows us to unpack their values with tuple assignment, much as we did for zip:
>>>E = enumerate(S)>>>E<enumerate object at 0x0000000002A8B900> >>>next(E)(0, 's') >>>next(E)(1, 'p') >>>next(E)(2, 'a')
We don’t normally see this machinery because all iteration contexts—including list comprehensions, the subject of Chapter 14—run the iteration protocol automatically:
>>>[c * i for (i, c) in enumerate(S)]['', 'p', 'aa', 'mmm'] >>>for (i, l) in enumerate(open('test.txt')):... print('%s) %s' % (i, l.rstrip()))... 0) aaaaaa 1) bbbbbb 2) cccccc
To fully understand iteration concepts like enumerate, zip, and list comprehensions, though, we need to move on to the next chapter for a more formal dissection.