Although the __getitem__ technique of the prior section works, it’s really just a fallback for iteration. Today, all iteration contexts in Python will try the __iter__ method first, before trying __getitem__. That is, they prefer the iteration protocol we learned about in Chapter 14 to repeatedly indexing an object; only if the object does not support the iteration protocol is indexing attempted instead. Generally speaking, you should prefer __iter__ too—it supports general iteration contexts better than __getitem__ can.
Technically, iteration contexts work by passing an iterable object to the iter built-in function to invoke an __iter__ method, which is expected to return an iterator object. If it’s provided, Python then repeatedly calls this iterator object’s __next__ method to produce items until a StopIteration exception is raised. A next built-in function is also available as a convenience for manual iterations—next(I) is the same as I.__next__(). For a review of this model’s essentials, see Figure 14-1 in Chapter 14.
This iterable object interface is given priority and attempted first. Only if no such __iter__ method is found, Python falls back on the __getitem__ scheme and repeatedly indexes by offsets as before, until an IndexError exception is raised.
Version skew note: As described in Chapter 14, if you are using Python 2.X, the I.__next__() iterator method just described is named I.next() in your Python, and the next(I) built-in is present for portability—it calls I.next() in 2.X and I.__next__() in 3.X. Iteration works the same in 2.X in all other respects.
In the __iter__ scheme, classes implement user-defined iterables by simply implementing the iteration protocol introduced in Chapter 14 and elaborated in Chapter 20. For example, the following file uses a class to define a user-defined iterable that generates squares on demand, instead of all at once (per the preceding note, in Python 2.X define next instead of __next__, and print with a trailing comma as usual):
# File squares.py class Squares: def __init__(self, start, stop): # Save state when created self.value = start - 1 self.stop = stop def __iter__(self): # Get iterator object on iter return self def __next__(self): # Return a square on each iteration if self.value == self.stop: # Also called by next built-in raise StopIteration self.value += 1 return self.value ** 2
When imported, its instances can appear in iteration contexts just like built-ins:
%python>>>from squares import Squares>>>for i in Squares(1, 5):# for calls iter, which calls __iter__print(i, end=' ')# Each iteration calls __next__ 1 4 9 16 25
Here, the iterator object returned by __iter__ is simply the instance self, because the __next__ method is part of this class itself. In more complex scenarios, the iterator object may be defined as a separate class and object with its own state information to support multiple active iterations over the same data (we’ll see an example of this in a moment). The end of the iteration is signaled with a Python raise statement—introduced in Chapter 29 and covered in full in the next part of this book, but which simply raises an exception as if Python itself had done so. Manual iterations work the same on user-defined iterables as they do on built-in types as well:
>>>X = Squares(1, 5)# Iterate manually: what loops do >>>I = iter(X)# iter calls __iter__ >>>next(I)# next calls __next__ (in 3.X) 1 >>>next(I)4...more omitted...>>>next(I)25 >>>next(I)# Can catch this in try statement StopIteration
An equivalent coding of this iterable with __getitem__ might be less natural, because the for would then iterate through all offsets zero and higher; the offsets passed in would be only indirectly related to the range of values produced (0..N would need to map to start..stop). Because __iter__ objects retain explicitly managed state between next calls, they can be more general than __getitem__.
On the other hand, iterables based on __iter__ can sometimes be more complex and less functional than those based on __getitem__. They are really designed for iteration, not random indexing—in fact, they don’t overload the indexing expression at all, though you can collect their items in a sequence such as a list to enable other operations:
>>>X = Squares(1, 5)>>>X[1]TypeError: 'Squares' object does not support indexing >>>list(X)[1]4
The __iter__ scheme is also the implementation for all the other iteration contexts we saw in action for the __getitem__ method—membership tests, type constructors, sequence assignment, and so on. Unlike our prior __getitem__ example, though, we also need to be aware that a class’s __iter__ may be designed for a single traversal only, not many. Classes choose scan behavior explicitly in their code.
For example, because the current Squares class’s __iter__ always returns self with just one copy of iteration state, it is a one-shot iteration; once you’ve iterated over an instance of that class, it’s empty. Calling __iter__ again on the same instance returns self again, in whatever state it may have been left. You generally need to make a new iterable instance object for each new iteration:
>>>X = Squares(1, 5)# Make an iterable with state >>>[n for n in X]# Exhausts items: __iter__ returns self [1, 4, 9, 16, 25] >>>[n for n in X]# Now it's empty: __iter__ returns same self [] >>>[n for n in Squares(1, 5)]# Make a new iterable object [1, 4, 9, 16, 25] >>>list(Squares(1, 3))# A new object for each new __iter__ call [1, 4, 9]
To support multiple iterations more directly, we could also recode this example with an extra class or other technique, as we will in a moment. As is, though, by creating a new instance for each iteration, you get a fresh copy of iteration state:
>>>36 in Squares(1, 10)# Other iteration contexts True >>>a, b, c = Squares(1, 3)# Each calls __iter__ and then __next__ >>>a, b, c(1, 4, 9) >>>':'.join(map(str, Squares(1, 5)))'1:4:9:16:25'
Just like single-scan built-ins such as map, converting to a list supports multiple scans as well, but adds time and space performance costs, which may or may not be significant to a given program:
>>>X = Squares(1, 5)>>>tuple(X), tuple(X)# Iterator exhausted in second tuple() ((1, 4, 9, 16, 25), ()) >>>X = list(Squares(1, 5))>>>tuple(X), tuple(X)((1, 4, 9, 16, 25), (1, 4, 9, 16, 25))
We’ll improve this to support multiple scans more directly ahead, after a bit of compare-and-contrast.
Notice that the preceding example would probably be simpler if it was coded with generator functions or expressions—tools introduced in Chapter 20 that automatically produce iterable objects and retain local variable state between iterations:
>>>def gsquares(start, stop):for i in range(start, stop + 1):yield i ** 2>>>for i in gsquares(1, 5):print(i, end=' ')1 4 9 16 25 >>>for i in (x ** 2 for x in range(1, 6)):print(i, end=' ')1 4 9 16 25
Unlike classes, generator functions and expressions implicitly save their state and create the methods required to conform to the iteration protocol—with obvious advantages in code conciseness for simpler examples like these. On the other hand, the class’s more explicit attributes and methods, extra structure, inheritance hierarchies, and support for multiple behaviors may be better suited for richer use cases.
Of course, for this artificial example, you could in fact skip both techniques and simply use a for loop, map, or a list comprehension to build the list all at once. Barring performance data to the contrary, the best and fastest way to accomplish a task in Python is often also the simplest:
>>> [x ** 2 for x in range(1, 6)]
[1, 4, 9, 16, 25]
However, classes may be better at modeling more complex iterations, especially when they can benefit from the assets of classes in general. An iterable that produces items in a complex database or web service result, for example, might be able to take fuller advantage of classes. The next section explores another use case for classes in user-defined iterables.
Earlier, I mentioned that the iterator object (with a __next__) produced by an iterable may be defined as a separate class with its own state information to more directly support multiple active iterations over the same data. Consider what happens when we step across a built-in type like a string:
>>>S = 'ace'>>>for x in S:for y in S:print(x + y, end=' ')aa ac ae ca cc ce ea ec ee
Here, the outer loop grabs an iterator from the string by calling iter, and each nested loop does the same to get an independent iterator. Because each active iterator has its own state information, each loop can maintain its own position in the string, regardless of any other active loops. Moreover, we’re not required to make a new string or convert to a list each time; the single string object itself supports multiple scans.
We saw related examples earlier, in Chapter 14 and Chapter 20. For instance, generator functions and expressions, as well as built-ins like map and zip, proved to be single-iterator objects, thus supporting a single active scan. By contrast, the range built-in, and other built-in types like lists, support multiple active iterators with independent positions.
When we code user-defined iterables with classes, it’s up to us to decide whether we will support a single active iteration or many. To achieve the multiple-iterator effect, __iter__ simply needs to define a new stateful object for the iterator, instead of returning self for each iterator request.
The following SkipObject class, for example, defines an iterable object that skips every other item on iterations. Because its iterator object is created anew from a supplemental class for each iteration, it supports multiple active loops directly (this is file skipper.py in the book’s examples):
#!python3 # File skipper.py class SkipObject: def __init__(self, wrapped): # Save item to be used self.wrapped = wrapped def __iter__(self): return SkipIterator(self.wrapped) # New iterator each time class SkipIterator: def __init__(self, wrapped): self.wrapped = wrapped # Iterator state information self.offset = 0 def __next__(self): if self.offset >= len(self.wrapped): # Terminate iterations raise StopIteration else: item = self.wrapped[self.offset] # else return and skip self.offset += 2 return item if __name__ == '__main__': alpha = 'abcdef' skipper = SkipObject(alpha) # Make container object I = iter(skipper) # Make an iterator on it print(next(I), next(I), next(I)) # Visit offsets 0, 2, 4 for x in skipper: # for calls __iter__ automatically for y in skipper: # Nested fors call __iter__ again each time print(x + y, end=' ') # Each iterator has its own state, offset
A quick portability note: as is, this is 3.X-only code. To make it 2.X compatible, import the 3.X print function, and either use next instead of __next__ for 2.X-only use, or alias the two names in the class’s scope for dual 2.X/3.X usage (file skipper_2x.py in the book’s examples does):
#!python
from __future__ import print_function # 2.X/3.X compatibility
...
class SkipIterator:
...
def __next__(self):
...
next = __next__ # 2.X/3.X compatibility
When the appropriate version is run in either Python, this example works like the nested loops with built-in strings. Each active loop has its own position in the string because each obtains an independent iterator object that records its own state information:
% python skipper.py
a c e
aa ac ae ca cc ce ea ec ee
By contrast, our earlier Squares example supports just one active iteration, unless we call Squares again in nested loops to obtain new objects. Here, there is just one SkipObject iterable, with multiple iterator objects created from it.
As before, we could achieve similar results with built-in tools—for example, slicing with a third bound to skip items:
>>>S = 'abcdef'>>>for x in S[::2]:for y in S[::2]:# New objects on each iterationprint(x + y, end=' ')aa ac ae ca cc ce ea ec ee
This isn’t quite the same, though, for two reasons. First, each slice expression here will physically store the result list all at once in memory; iterables, on the other hand, produce just one value at a time, which can save substantial space for large result lists. Second, slices produce new objects, so we’re not really iterating over the same object in multiple places here. To be closer to the class, we would need to make a single object to step across by slicing ahead of time:
>>>S = 'abcdef'>>>S = S[::2]>>>S'ace' >>>for x in S:for y in S:# Same object, new iteratorsprint(x + y, end=' ')aa ac ae ca cc ce ea ec ee
This is more similar to our class-based solution, but it still stores the slice result in memory all at once (there is no generator form of built-in slicing today), and it’s only equivalent for this particular case of skipping every other item.
Because user-defined iterables coded with classes can do anything a class can do, they are much more general than this example may imply. Though such generality is not required in all applications, user-defined iterables are a powerful tool—they allow us to make arbitrary objects look and feel like the other sequences and iterables we have met in this book. We could use this technique with a database object, for example, to support iterations over large database fetches, with multiple cursors into the same query result.
And now, for something completely implicit—but potentially useful nonetheless. In some applications, it’s possible to minimize coding requirements for user-defined iterables by combining the __iter__ method we’re exploring here and the yield generator function statement we studied in Chapter 20. Because generator functions automatically save local variable state and create required iterator methods, they fit this role well, and complement the state retention and other utility we get from classes.
As a review, recall that any function that contains a yield statement is turned into a generator function. When called, it returns a new generator object with automatic retention of local scope and code position, an automatically created __iter__ method that simply returns itself, and an automatically created __next__ method (next in 2.X) that starts the function or resumes it where it last left off:
>>>def gen(x):for i in range(x): yield i ** 2>>>G = gen(5)# Create a generator with __iter__ and __next__ >>>G.__iter__() == G# Both methods exist on the same object True >>>I = iter(G)# Runs __iter__: generator returns itself >>>next(I), next(I)# Runs __next__ (next in 2.X) (0, 1) >>>list(gen(5))# Iteration contexts automatically run iter and next [0, 1, 4, 9, 16]
This is still true even if the generator function with a yield happens to be a method named __iter__: whenever invoked by an iteration context tool, such a method will return a new generator object with the requisite __next__. As an added bonus, generator functions coded as methods in classes have access to saved state in both instance attributes and local scope variables.
For example, the following class is equivalent to the initial Squares user-defined iterable we coded earlier in squares.py.
# File squares_yield.py class Squares: # __iter__ + yield generator def __init__(self, start, stop): # __next__ is automatic/implied self.start = start self.stop = stop def __iter__(self): for value in range(self.start, self.stop + 1): yield value ** 2
There’s no need to alias next to __next__ for 2.X compatibility here, because this method is now automated and implied by the use of yield. As before, for loops and other iteration tools iterate through instances of this class automatically:
%python>>>from squares_yield import Squares>>>for i in Squares(1, 5): print(i, end=' ')1 4 9 16 25
And as usual, we can look under the hood to see how this actually works in iteration contexts. Running our class instance through iter obtains the result of calling __iter__ as usual, but in this case the result is a generator object with an automatically created __next__ of the same sort we always get when calling a generator function that contains a yield. The only difference here is that the generator function is automatically called on iter. Invoking the result object’s next interface produces results on demand:
>>>S = Squares(1, 5)# Runs __init__: class saves instance state >>>S<squares_yield.Squares object at 0x000000000294B630> >>>I = iter(S)# Runs __iter__: returns a generator >>>I<generator object __iter__ at 0x00000000029A8CF0> >>>next(I)1 >>>next(I)# Runs generator's __next__ 4...etc...>>>next(I)# Generator has both instance and local scope state StopIteration
It may also help to notice that we could name the generator method something other than __iter__ and call manually to iterate—Squares(1,5).gen(), for example. Using the __iter__ name invoked automatically by iteration tools simply skips a manual attribute fetch and call step:
class Squares: # Non __iter__ equivalent (squares_manual.py) def __init__(...): ... def gen(self): for value in range(self.start, self.stop + 1): yield value ** 2 %python>>>from squares_manual import Squares>>>for i in Squares(1, 5).gen(): print(i, end=' ')...same results... >>>S = Squares(1, 5)>>>I = iter(S.gen())# Call generator manually for iterable/iterator >>>next(I)...same results...
Coding the generator as __iter__ instead cuts out the middleman in your code, though both schemes ultimately wind up creating a new generator object for each iteration:
With __iter__, iteration triggers __iter__, which returns a new generator with __next__.
Without __iter__, your code calls to make a generator, which returns itself for __iter__.
See Chapter 20 for more on yield and generators if this is puzzling, and compare it with the more explicit __next__ version in squares.py earlier. You’ll notice that this new squares_yield.py version is 4 lines shorter (7 versus 11). In a sense, this scheme reduces class coding requirements much like the closure functions of Chapter 17, but in this case does so with a combination of functional and OOP techniques, instead of an alternative to classes. For example, the generator method still leverages self attributes.
This may also very well seem like one too many levels of magic to some observers—it relies on both the iteration protocol and the object creation of generators, both of which are highly implicit (in contradiction of longstanding Python themes: see import this). Opinions aside, it’s important to understand the non-yield flavor of class iterables too, because it’s explicit, general, and sometimes broader in scope.
Still, the __iter__/yield technique may prove effective in cases where it applies. It also comes with a substantial advantage—as the next section explains.
Besides its code conciseness, the user-defined class iterable of the prior section based upon the __iter__/yield combination has an important added bonus—it also supports multiple active iterators automatically. This naturally follows from the fact that each call to __iter__ is a call to a generator function, which returns a new generator with its own copy of the local scope for state retention:
%python>>>from squares_yield import Squares# Using the __iter__/yield Squares >>>S = Squares(1, 5)>>>I = iter(S)>>>next(I); next(I)1 4 >>>J = iter(S)# With yield, multiple iterators automatic >>>next(J)1 >>>next(I)# I is independent of J: own local state 9
Although generator functions are single-scan iterables, the implicit calls to __iter__ in iteration contexts make new generators supporting new independent scans:
>>>S = Squares(1, 3)>>>for i in S:# Each for calls __iter__for j in S:print('%s:%s' % (i, j), end=' ')1:1 1:4 1:9 4:1 4:4 4:9 9:1 9:4 9:9
To do the same without yield requires a supplemental class that stores iterator state explicitly and manually, using techniques of the preceding section (and grows to 15 lines: 8 more than with yield):
# File squares_nonyield.py class Squares: def __init__(self, start, stop): # Non-yield generator self.start = start # Multiscans: extra object self.stop = stop def __iter__(self): return SquaresIter(self.start, self.stop) class SquaresIter: def __init__(self, start, stop): self.value = start - 1 self.stop = stop def __next__(self): if self.value == self.stop: raise StopIteration self.value += 1 return self.value ** 2
This works the same as the yield multiscan version, but with more, and more explicit, code:
%python>>>from squares_nonyield import Squares>>>for i in Squares(1, 5): print(i, end=' ')1 4 9 16 25 >>> >>>S = Squares(1, 5)>>>I = iter(S)>>>next(I); next(I)1 4 >>>J = iter(S)# Multiple iterators without yield >>>next(J)1 >>>next(I)9 >>>S = Squares(1, 3)>>>for i in S:# Each for calls __iter___for j in S:print('%s:%s' % (i, j), end=' ')1:1 1:4 1:9 4:1 4:4 4:9 9:1 9:4 9:9
Finally, the generator-based approach could similarly remove the need for an extra iterator class in the prior item-skipper example of file skipper.py, thanks to its automatic methods and local variable state retention (and checks in at 9 lines versus the original’s 16):
# File skipper_yield.py class SkipObject: # Another __iter__ + yield generator def __init__(self, wrapped): # Instance scope retained normally self.wrapped = wrapped # Local scope state saved auto def __iter__(self): offset = 0 while offset < len(self.wrapped): item = self.wrapped[offset] offset += 2 yield item
This works the same as the non-yield multiscan version, but with less, and less explicit, code:
%python>>>from skipper_yield import SkipObject>>>skipper = SkipObject('abcdef')>>>I = iter(skipper)>>>next(I); next(I); next(I)'a' 'c' 'e' >>>for x in skipper:# Each for calls __iter__: new auto generatorfor y in skipper:print(x + y, end=' ')aa ac ae ca cc ce ea ec ee
Of course, these are all artificial examples that could be replaced with simpler tools like comprehensions, and their code may or may not scale up in kind to more realistic tasks. Study these alternatives to see how they compare. As so often in programming, the best tool for the job will likely be the best tool for your job!