Comprehension Syntax Summary

>>> [x * x for x in range(10)]            # List comprehension: builds list
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]      # Like list(generator expr)

>>> (x * x for x in range(10))            # Generator expression: produces items
<generator object at 0x009E7328>          # Parens are often optional

>>> {x * x for x in range(10)}            # Set comprehension, 3.X and 2.7
{0, 1, 4, 81, 64, 9, 16, 49, 25, 36}      # {x, y} is a set in these versions too

>>> {x: x * x for x in range(10)}         # Dictionary comprehension, 3.X and 2.7
{0: 0, 1: 1, 2: 4, 3: 9, 4: 16, 5: 25, 6: 36, 7: 49, 8: 64, 9: 81}

Now that we’ve seen all comprehension forms, be sure to also review Chapter 17’s overview of the localization of loop variables in these expressions. Python 3.X localizes loop variables in all four forms—temporary loop variable names in generator, set, dictionary, and list comprehensions are local to the expression. They don’t clash with names outside, but are also not available there, and work differently than the for loop iteration statement:

c:\code> py −3
>>> (X for X in range(5))
<generator object <genexpr> at 0x00000000028E4798>
>>> X
NameError: name 'X' is not defined

>>> X = 99
>>> [X for X in range(5)]         # 3.X: generator, set, dict, and list localize
[0, 1, 2, 3, 4]
>>> X
99

>>> Y = 99
>>> for Y in range(5): pass       # But loop statements do not localize names

>>> Y
4

As mentioned in Chapter 17, 3.X variables assigned in a comprehension are really a further nested special-case scope; other names referenced within these expressions follow the usual LEGB rules. In the following generator, for example, Z is localized in the comprehension, but Y and X are found in the enclosing local and global scopes as usual:

>>> X = 'aaa'
>>> def func():
        Y = 'bbb'
        print(''.join(Z for Z in X + Y))       # Z comprehension, Y local, X global

>>> func()
aaabbb

Python 2.X is the same in this regard, except that list comprehension variables are not localized—they work just like for loops and keep their last iteration values, but are also open to unexpected clashes with outside names. Generator, set, and dictionary forms localize names as in 3.X:

c:\code> py −2
>>> (X for X in range(5))
<generator object <genexpr> at 0x0000000002147EE8>
>>> X
NameError: name 'X' is not defined

>>> X = 99
>>> [X for X in range(5)]         # 2.X: List does not localize its names, like for
[0, 1, 2, 3, 4]
>>> X
4

>>> Y = 99
>>> for Y in range(5): pass       # for loops do not localize names in 2.X or 3.X

>>> Y
4

If you care about version portability, and symmetry with the for loop statement, use unique names for variables in comprehension expressions as a rule of thumb. The 2.X behavior makes sense given that a generator object is discarded after it finishes producing results, but a list comprehension is equivalent to a for loop—though this analogy doesn’t hold for the set and dictionary forms that localize their names in both Pythons, and are, somewhat coincidentally, the topic of the next section.

In a sense, set and dictionary comprehensions are just syntactic sugar for passing generator expressions to the type names. Because both accept any iterable, a generator works well here:

>>> {x * x for x in range(10)}                # Comprehension
{0, 1, 4, 81, 64, 9, 16, 49, 25, 36}
>>> set(x * x for x in range(10))             # Generator and type name
{0, 1, 4, 81, 64, 9, 16, 49, 25, 36}

>>> {x: x * x for x in range(10)}
{0: 0, 1: 1, 2: 4, 3: 9, 4: 16, 5: 25, 6: 36, 7: 49, 8: 64, 9: 81}
>>> dict((x, x * x) for x in range(10))
{0: 0, 1: 1, 2: 4, 3: 9, 4: 16, 5: 25, 6: 36, 7: 49, 8: 64, 9: 81}

>>> x                                         # Loop variable localized in 2.X + 3.X
NameError: name 'x' is not defined

As for list comprehensions, though, we can always build the result objects with manual code, too. Here are statement-based equivalents of the last two comprehensions (though they differ in that name localization):

>>> res = set()
>>> for x in range(10):                        # Set comprehension equivalent
        res.add(x * x)

>>> res
{0, 1, 4, 81, 64, 9, 16, 49, 25, 36}

>>> res = {}
>>> for x in range(10):                        # Dict comprehension equivalent
        res[x] = x * x

>>> res
{0: 0, 1: 1, 2: 4, 3: 9, 4: 16, 5: 25, 6: 36, 7: 49, 8: 64, 9: 81}

>>> x   # Localized in comprehension expressions, but not in loop statements
9

Notice that although both set and dictionary comprehensions accept and scan iterables, they have no notion of generating results on demand—both forms build complete objects all at once. If you mean to produce keys and values upon request, a generator expression is more appropriate:

>>> G = ((x, x * x) for x in range(10))
>>> next(G)
(0, 0)
>>> next(G)
(1, 1)

Like list comprehensions and generator expressions, both set and dictionary comprehensions support nested associated if clauses to filter items out of the result—the following collect squares of even items (i.e., items having no remainder for division by 2) in a range:

>>> [x * x for x in range(10) if x % 2 == 0]           # Lists are ordered
[0, 4, 16, 36, 64]
>>> {x * x for x in range(10) if x % 2 == 0}           # But sets are not
{0, 16, 4, 64, 36}
>>> {x: x * x for x in range(10) if x % 2 == 0}        # Neither are dict keys
{0: 0, 8: 64, 2: 4, 4: 16, 6: 36}

Nested for loops work as well, though the unordered and no-duplicates nature of both types of objects can make the results a bit less straightforward to decipher:

>>> [x + y for x in [1, 2, 3] for y in [4, 5, 6]]      # Lists keep duplicates
[5, 6, 7, 6, 7, 8, 7, 8, 9]
>>> {x + y for x in [1, 2, 3] for y in [4, 5, 6]}      # But sets do not
{8, 9, 5, 6, 7}
>>> {x: y for x in [1, 2, 3] for y in [4, 5, 6]}       # Neither do dict keys
{1: 6, 2: 6, 3: 6}

Like list comprehensions, the set and dictionary varieties can also iterate over any type of iterable—lists, strings, files, ranges, and anything else that supports the iteration protocol:

>>> {x + y for x in 'ab' for y in 'cd'}
{'ac', 'bd', 'bc', 'ad'}

>>> {x + y: (ord(x), ord(y)) for x in 'ab' for y in 'cd'}
{'ac': (97, 99), 'bd': (98, 100), 'bc': (98, 99), 'ad': (97, 100)}

>>> {k * 2 for k in ['spam', 'ham', 'sausage'] if k[0] == 's'}
{'sausagesausage', 'spamspam'}

>>> {k.upper(): k * 2 for k in ['spam', 'ham', 'sausage'] if k[0] == 's'}
{'SAUSAGE': 'sausagesausage', 'SPAM': 'spamspam'}

For more details, experiment with these tools on your own. They may or may not have a performance advantage over the generator or for loop alternatives, but we would have to time their performance explicitly to be sure—which seems a natural segue to the next chapter.