New Iterables in Python 3.X

Although this saves memory space, it can impact your coding styles in some contexts. In various places in this book so far, for example, we’ve had to wrap up some function and method call results in a list(...) call in order to force them to produce all their results at once for display:

>>> zip('abc', 'xyz')                  # An iterable in Python 3.X (a list in 2.X)
<zip object at 0x000000000294C308>

>>> list(zip('abc', 'xyz'))            # Force list of results in 3.X to display
[('a', 'x'), ('b', 'y'), ('c', 'z')]

A similar conversion is required if we wish to apply list or sequence operations to most iterables that generate items on demand—to index, slice, or concatenate the iterable itself, for example. The list results for these tools in 2.X support such operations directly:

>>> Z = zip((1, 2), (3, 4))            # Unlike 2.X lists, cannot index, etc.
>>> Z[0]
TypeError: 'zip' object is not subscriptable

As we’ll see in more detail in Chapter 20, conversion to lists may also be more subtly required to support multiple iterations for newly iterable tools that support just one scan such as map and zip—unlike their 2.X list forms, their values in 3.X are exhausted after a single pass:

>>> M = map(lambda x: 2 ** x, range(3))
>>> for i in M: print(i)
...
1
2
4
>>> for i in M: print(i)               # Unlike 2.X lists, one pass only (zip too)
...
>>>

Such conversion isn’t required in 2.X, because functions like zip return lists of results. In 3.X, though, they return iterable objects, producing results on demand. This may break 2.X code, and means extra typing is required to display the results at the interactive prompt (and possibly in some other contexts), but it’s an asset in larger programs—delayed evaluation like this conserves memory and avoids pauses while large result lists are computed. Let’s take a quick look at some of the new 3.X iterables in action.

We studied the range built-in’s basic behavior in the preceding chapter. In 3.X, it returns an iterable that generates numbers in the range on demand, instead of building the result list in memory. This subsumes the older 2.X xrange (see the upcoming version skew note), and you must use list(range(...)) to force an actual range list if one is needed (e.g., to display results):

C:\code> c:\python33\python
>>> R = range(10)                # range returns an iterable, not a list
>>> R
range(0, 10)

>>> I = iter(R)                  # Make an iterator from the range iterable
>>> next(I)                      # Advance to next result
0                                # What happens in for loops, comprehensions, etc.
>>> next(I)
1
>>> next(I)
2

>>> list(range(10))              # To force a list if required
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

Unlike the list returned by this call in 2.X, range objects in 3.X support only iteration, indexing, and the len function. They do not support any other sequence operations (use list(...) if you require more list tools):

>>> len(R)                       # range also does len and indexing, but no others
10
>>> R[0]
0
>>> R[-1]
9

>>> next(I)                      # Continue taking from iterator, where left off
3
>>> I.__next__()                 # .next() becomes .__next__(), but use new next()
4

Like range, the map, zip, and filter built-ins also become iterables in 3.X to conserve space, rather than producing a result list all at once in memory. All three not only process iterables, as in 2.X, but also return iterable results in 3.X. Unlike range, though, they are their own iterators—after you step through their results once, they are exhausted. In other words, you can’t have multiple iterators on their results that maintain different positions in those results.

Here is the case for the map built-in we met in the prior chapter. As with other iterables, you can force a list with list(...) if you really need one, but the default behavior can save substantial space in memory for large result sets:

>>> M = map(abs, (-1, 0, 1))            # map returns an iterable, not a list
>>> M
<map object at 0x00000000029B75C0>
>>> next(M)                             # Use iterator manually: exhausts results
1                                       # These do not support len() or indexing
>>> next(M)
0
>>> next(M)
1
>>> next(M)
StopIteration

>>> for x in M: print(x)                # map iterator is now empty: one pass only
...

>>> M = map(abs, (-1, 0, 1))            # Make a new iterable/iterator to scan again
>>> for x in M: print(x)                # Iteration contexts auto call next()
...
1
0
1
>>> list(map(abs, (-1, 0, 1)))          # Can force a real list if needed
[1, 0, 1]

The zip built-in, introduced in the prior chapter, is an iteration context itself, but also returns an iterable with an iterator that works the same way:

>>> Z = zip((1, 2, 3), (10, 20, 30))    # zip is the same: a one-pass iterator
>>> Z
<zip object at 0x0000000002951108>

>>> list(Z)
[(1, 10), (2, 20), (3, 30)]

>>> for pair in Z: print(pair)          # Exhausted after one pass
...

>>> Z = zip((1, 2, 3), (10, 20, 30))
>>> for pair in Z: print(pair)          # Iterator used automatically or manually
...
(1, 10)
(2, 20)
(3, 30)

>>> Z = zip((1, 2, 3), (10, 20, 30))    # Manual iteration (iter() not needed)
>>> next(Z)
(1, 10)
>>> next(Z)
(2, 20)

The filter built-in, which we met briefly in Chapter 12 and will study in the next part of this book, is also analogous. It returns items in an iterable for which a passed-in function returns True (as we’ve learned, in Python True includes nonempty objects, and bool returns an object’s truth value):

>>> filter(bool, ['spam', '', 'ni'])
<filter object at 0x00000000029B7B70>
>>> list(filter(bool, ['spam', '', 'ni']))
['spam', 'ni']

Like most of the tools discussed in this section, filter both accepts an iterable to process and returns an iterable to generate results in 3.X. It can also generally be emulated by extended list comprehension syntax that automatically tests truth values:

>>> [x for x in ['spam', '', 'ni'] if bool(x)]
['spam', 'ni']
>>> [x for x in ['spam', '', 'ni'] if x]
['spam', 'ni']

It’s important to see how the range object differs from the built-ins described in this section—it supports len and indexing, it is not its own iterator (you make one with iter when iterating manually), and it supports multiple iterators over its result that remember their positions independently:

>>> R = range(3)                           # range allows multiple iterators
>>> next(R)
TypeError: range object is not an iterator

>>> I1 = iter(R)
>>> next(I1)
0
>>> next(I1)
1
>>> I2 = iter(R)                           # Two iterators on one range
>>> next(I2)
0
>>> next(I1)                               # I1 is at a different spot than I2
2

By contrast, in 3.X zip, map, and filter do not support multiple active iterators on the same result; because of this the iter call is optional for stepping through such objects’ results—their iter is themselves (in 2.X these built-ins return multiple-scan lists so the following does not apply):

>>> Z = zip((1, 2, 3), (10, 11, 12))
>>> I1 = iter(Z)
>>> I2 = iter(Z)                           # Two iterators on one zip
>>> next(I1)
(1, 10)
>>> next(I1)
(2, 11)
>>> next(I2)                               # (3.X) I2 is at same spot as I1!
(3, 12)

>>> M = map(abs, (-1, 0, 1))               # Ditto for map (and filter)
>>> I1 = iter(M); I2 = iter(M)
>>> print(next(I1), next(I1), next(I1))
1 0 1
>>> next(I2)                               # (3.X) Single scan is exhausted!
StopIteration

>>> R = range(3)                           # But range allows many iterators
>>> I1, I2 = iter(R), iter(R)
>>> [next(I1), next(I1), next(I1)]
[0 1 2]
>>> next(I2)                               # Multiple active scans, like 2.X lists
0

When we code our own iterable objects with classes later in the book (Chapter 30), we’ll see that multiple iterators are usually supported by returning new objects for the iter call; a single iterator generally means an object returns itself. In Chapter 20, we’ll also find that generator functions and expressions behave like map and zip instead of range in this regard, supporting just a single active iteration scan. In that chapter, we’ll see some subtle implications of one-shot iterators in loops that attempt to scan multiple times—code that formerly treated these as lists may fail without manual list conversions.

Finally, as we saw briefly in Chapter 8, in Python 3.X the dictionary keys, values, and items methods return iterable view objects that generate result items one at a time, instead of producing result lists all at once in memory. Views are also available in 2.7 as an option, but under special method names to avoid impacting existing code. View items maintain the same physical ordering as that of the dictionary and reflect changes made to the underlying dictionary. Now that we know more about iterables here’s the rest of this story—in Python 3.3 (your key order may vary):

>>> D = dict(a=1, b=2, c=3)
>>> D
{'a': 1, 'b': 2, 'c': 3}

>>> K = D.keys()                              # A view object in 3.X, not a list
>>> K
dict_keys(['a', 'b', 'c'])

>>> next(K)                                   # Views are not iterators themselves
TypeError: dict_keys object is not an iterator

>>> I = iter(K)                               # View iterables have an iterator,
>>> next(I)                                   # which can be used manually,
'a'                                           # but does not support len(), index
>>> next(I)
'b'

>>> for k in D.keys(): print(k, end=' ')      # All iteration contexts use auto
...
a b c

As for all iterables that produce values on request, you can always force a 3.X dictionary view to build a real list by passing it to the list built-in. However, this usually isn’t required except to display results interactively or to apply list operations like indexing:

>>> K = D.keys()
>>> list(K)                              # Can still force a real list if needed
['a', 'b', 'c']

>>> V = D.values()                       # Ditto for values() and items() views
>>> V
dict_values([1, 2, 3])
>>> list(V)                              # Need list() to display or index as list
[1, 2, 3]

>>> V[0]
TypeError: 'dict_values' object does not support indexing
>>> list(V)[0]
1

>>> list(D.items())
[('a', 1), ('b', 2), ('c', 3)]

>>> for (k, v) in D.items(): print(k, v, end=' ')
...
a 1 b 2 c 3

In addition, 3.X dictionaries still are iterables themselves, with an iterator that returns successive keys. Thus, it’s not often necessary to call keys directly in this context:

>>> D                                    # Dictionaries still produce an iterator
{'a': 1, 'b': 2, 'c': 3}                 # Returns next key on each iteration
>>> I = iter(D)
>>> next(I)
'a'
>>> next(I)
'b'

>>> for key in D: print(key, end=' ')    # Still no need to call keys() to iterate
...                                      # But keys is an iterable in 3.X too!
a b c

Finally, remember again that because keys no longer returns a list, the traditional coding pattern for scanning a dictionary by sorted keys won’t work in 3.X. Instead, convert keys views first with a list call, or use the sorted call on either a keys view or the dictionary itself, as follows. We saw this in Chapter 8, but it’s important enough to 2.X programmers making the switch to demonstrate again:

>>> D
{'a': 1, 'b': 2, 'c': 3}
>>> for k in sorted(D.keys()): print(k, D[k], end=' ')
...
a 1 b 2 c 3
>>> for k in sorted(D): print(k, D[k], end=' ')    # "Best practice" key sorting
...
a 1 b 2 c 3