Perhaps the best way to understand lists is to see them at work. Let’s once again turn to some simple interpreter interactions to illustrate the operations in Table 8-1.
Because they are sequences, lists support many of the same operations as strings. For example, lists respond to the + and * operators much like strings—they mean concatenation and repetition here too, except that the result is a new list, not a string:
%python>>>len([1, 2, 3])# Length 3 >>>[1, 2, 3] + [4, 5, 6]# Concatenation [1, 2, 3, 4, 5, 6] >>>['Ni!'] * 4# Repetition ['Ni!', 'Ni!', 'Ni!', 'Ni!']
Although the + operator works the same for lists and strings, it’s important to know that it expects the same sort of sequence on both sides—otherwise, you get a type error when the code runs. For instance, you cannot concatenate a list and a string unless you first convert the list to a string (using tools such as str or % formatting) or convert the string to a list (the list built-in function does the trick):
>>>str([1, 2]) + "34"# Same as "[1, 2]" + "34" '[1, 2]34' >>>[1, 2] + list("34")# Same as [1, 2] + ["3", "4"] [1, 2, '3', '4']
More generally, lists respond to all the sequence operations we used on strings in the prior chapter, including iteration tools:
>>>3 in [1, 2, 3]# Membership True >>>for x in [1, 2, 3]:...print(x, end=' ')# Iteration (2.X uses: print x,) ... 1 2 3
We will talk more formally about for iteration and the range built-ins of Table 8-1 in Chapter 13, because they are related to statement syntax. In short, for loops step through items in any sequence from left to right, executing one or more statements for each item; range produces successive integers.
The last items in Table 8-1, list comprehensions and map calls, are covered in more detail in Chapter 14 and expanded on in Chapter 20. Their basic operation is straightforward, though—as introduced in Chapter 4, list comprehensions are a way to build a new list by applying an expression to each item in a sequence (really, in any iterable), and are close relatives to for loops:
>>>res =[c * 4 for c in 'SPAM']# List comprehensions >>>res['SSSS', 'PPPP', 'AAAA', 'MMMM']
This expression is functionally equivalent to a for loop that builds up a list of results manually, but as we’ll learn in later chapters, list comprehensions are simpler to code and likely faster to run today:
>>>res = []>>>for c in 'SPAM':# List comprehension equivalent ...res.append(c * 4)... >>>res['SSSS', 'PPPP', 'AAAA', 'MMMM']
As also introduced briefly in Chapter 4, the map built-in function does similar work, but applies a function to items in a sequence and collects all the results in a new list:
>>> list(map(abs, [−1, −2, 0, 1, 2])) # Map a function across a sequence
[1, 2, 0, 1, 2]
Because we’re not quite ready for the full iteration story, we’ll postpone further details for now, but watch for a similar comprehension expression for dictionaries later in this chapter.
Because lists are sequences, indexing and slicing work the same way for lists as they do for strings. However, the result of indexing a list is whatever type of object lives at the offset you specify, while slicing a list always returns a new list:
>>>L = ['spam', 'Spam', 'SPAM!']>>>L[2]# Offsets start at zero 'SPAM!' >>>L[−2]# Negative: count from the right 'Spam' >>>L[1:]# Slicing fetches sections ['Spam', 'SPAM!']
One note here: because you can nest lists and other object types within lists, you will sometimes need to string together index operations to go deeper into a data structure. For example, one of the simplest ways to represent matrixes (multidimensional arrays) in Python is as lists with nested sublists. Here’s a basic 3 × 3 two-dimensional list-based array:
>>> matrix = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
With one index, you get an entire row (really, a nested sublist), and with two, you get an item within the row:
>>>matrix[1][4, 5, 6] >>>matrix[1][1]5 >>>matrix[2][0]7 >>>matrix = [[1, 2, 3],...[4, 5, 6],...[7, 8, 9]]>>>matrix[1][1]5
Notice in the preceding interaction that lists can naturally span multiple lines if you want them to because they are contained by a pair of brackets; the “...”s here are Python’s continuation line prompt (see Chapter 4 for comparable code without the “...”s, and watch for more on syntax in the next part of the book).
For more on matrixes, watch later in this chapter for a dictionary-based matrix representation, which can be more efficient when matrixes are largely empty. We’ll also continue this thread in Chapter 20 where we’ll write additional matrix code, especially with list comprehensions. For high-powered numeric work, the NumPy extension mentioned in Chapter 4 and Chapter 5 provides other ways to handle matrixes.
Because lists are mutable, they support operations that change a list object in place. That is, the operations in this section all modify the list object directly—overwriting its former value—without requiring that you make a new copy, as you had to for strings. Because Python deals only in object references, this distinction between changing an object in place and creating a new object matters; as discussed in Chapter 6, if you change an object in place, you might impact more than one reference to it at the same time.
When using a list, you can change its contents by assigning to either a particular item (offset) or an entire section (slice):
>>>L = ['spam', 'Spam', 'SPAM!']>>>L[1] = 'eggs'# Index assignment >>>L['spam', 'eggs', 'SPAM!'] >>>L[0:2] = ['eat', 'more']# Slice assignment: delete+insert >>>L# Replaces items 0,1 ['eat', 'more', 'SPAM!']
Both index and slice assignments are in-place changes—they modify the subject list directly, rather than generating a new list object for the result. Index assignment in Python works much as it does in C and most other languages: Python replaces the single object reference at the designated offset with a new one.
Slice assignment, the last operation in the preceding example, replaces an entire section of a list in a single step. Because it can be a bit complex, it is perhaps best thought of as a combination of two steps:
Deletion. The slice you specify to the left of the = is deleted.
Insertion. The new items contained in the iterable object to the right of the = are inserted into the list on the left, at the place where the old slice was deleted.[19]
This isn’t what really happens, but it can help clarify why the number of items inserted doesn’t have to match the number of items deleted. For instance, given a list L of two or more items, an assignment L[1:2]=[4,5] replaces one item with two—Python first deletes the one-item slice at [1:2] (from offset 1, up to but not including offset 2), then inserts both 4 and 5 where the deleted slice used to be.
This also explains why the second slice assignment in the following is really an insert—Python replaces an empty slice at [1:1] with two items; and why the third is really a deletion—Python deletes the slice (the item at offset 1), and then inserts nothing:
>>>L = [1, 2, 3]>>>L[1:2] = [4, 5]# Replacement/insertion >>>L[1, 4, 5, 3] >>>L[1:1] = [6, 7]# Insertion (replace nothing) >>>L[1, 6, 7, 4, 5, 3] >>>L[1:2] = []# Deletion (insert nothing) >>>L[1, 7, 4, 5, 3]
In effect, slice assignment replaces an entire section, or “column,” all at once—even if the column or its replacement is empty. Because the length of the sequence being assigned does not have to match the length of the slice being assigned to, slice assignment can be used to replace (by overwriting), expand (by inserting), or shrink (by deleting) the subject list. It’s a powerful operation, but frankly, one that you may not see very often in practice. There are often more straightforward and mnemonic ways to replace, insert, and delete (concatenation, and the insert, pop, and remove list methods, for example), which Python programmers tend to prefer in practice.
On the other hand, this operation can be used as a sort of in-place concatenation at the front of the list—per the next section’s method coverage, something the list’s extend does more mnemonically at list end:
>>>L = [1]>>>L[:0] = [2, 3, 4]# Insert all at :0, an empty slice at front >>>L[2, 3, 4, 1] >>>L[len(L):] = [5, 6, 7]# Insert all at len(L):, an empty slice at end >>>L[2, 3, 4, 1, 5, 6, 7] >>>L.extend([8, 9, 10])# Insert all at end, named method >>>L[2, 3, 4, 1, 5, 6, 7, 8, 9, 10]
Like strings, Python list objects also support type-specific method calls, many of which change the subject list in place:
>>>L = ['eat', 'more', 'SPAM!']>>>L.append('please')# Append method call: add item at end >>>L['eat', 'more', 'SPAM!', 'please'] >>>L.sort()# Sort list items ('S' < 'e') >>>L['SPAM!', 'eat', 'more', 'please']
Methods were introduced in Chapter 7. In brief, they are functions (really, object attributes that reference functions) that are associated with and act upon particular objects. Methods provide type-specific tools; the list methods presented here, for instance, are generally available only for lists.
Perhaps the most commonly used list method is append, which simply tacks a single item (object reference) onto the end of the list. Unlike concatenation, append expects you to pass in a single object, not a list. The effect of L.append(X) is similar to L+[X], but while the former changes L in place, the latter makes a new list.[20] The sort method orders the list’s items here, but merits a section of its own.
Another commonly seen method, sort, orders a list in place; it uses Python standard comparison tests (here, string comparisons, but applicable to every object type), and by default sorts in ascending order. You can modify sort behavior by passing in keyword arguments—a special “name=value” syntax in function calls that specifies passing by name and is often used for giving configuration options.
In sorts, the reverse argument allows sorts to be made in descending instead of ascending order, and the key argument gives a one-argument function that returns the value to be used in sorting—the string object’s standard lower case converter in the following (though its newer casefold may handle some types of Unicode text better):
>>>L = ['abc', 'ABD', 'aBe']>>>L.sort()# Sort with mixed case >>>L['ABD', 'aBe', 'abc'] >>>L = ['abc', 'ABD', 'aBe']>>>L.sort(key=str.lower)# Normalize to lowercase >>>L['abc', 'ABD', 'aBe'] >>> >>>L = ['abc', 'ABD', 'aBe']>>>L.sort(key=str.lower, reverse=True)# Change sort order >>>L['aBe', 'ABD', 'abc']
The sort key argument might also be useful when sorting lists of dictionaries, to pick out a sort key by indexing each dictionary. We’ll study dictionaries later in this chapter, and you’ll learn more about keyword function arguments in Part IV.
Comparison and sorts in 3.X: In Python 2.X, relative magnitude comparisons of differently typed objects (e.g., a string and a list) work—the language defines a fixed ordering among different types, which is deterministic, if not aesthetically pleasing. That is, the ordering is based on the names of the types involved: all integers are less than all strings, for example, because "int" is less than "str". Comparisons never automatically convert types, except when comparing numeric type objects.
In Python 3.X, this has changed: magnitude comparison of mixed types raises an exception instead of falling back on the fixed cross-type ordering. Because sorting uses comparisons internally, this means that [1, 2, 'spam'].sort() succeeds in Python 2.X but will raise an exception in Python 3.X. Sorting mixed-types fails by proxy.
Python 3.X also no longer supports passing in an arbitrary comparison function to sorts, to implement different orderings. The suggested workaround is to use the key=func keyword argument to code value transformations during the sort, and use the reverse=True keyword argument to change the sort order to descending. These were the typical uses of comparison functions in the past.
One warning here: beware that append and sort change the associated list object in place, but don’t return the list as a result (technically, they both return a value called None). If you say something like L=L.append(X), you won’t get the modified value of L (in fact, you’ll lose the reference to the list altogether!). When you use attributes such as append and sort, objects are changed as a side effect, so there’s no reason to reassign.
Partly because of such constraints, sorting is also available in recent Pythons as a built-in function, which sorts any collection (not just lists) and returns a new list for the result (instead of in-place changes):
>>>L = ['abc', 'ABD', 'aBe']>>>sorted(L, key=str.lower, reverse=True)# Sorting built-in ['aBe', 'ABD', 'abc'] >>>L = ['abc', 'ABD', 'aBe']>>>sorted([x.lower() for x in L], reverse=True)# Pretransform items: differs! ['abe', 'abd', 'abc']
Notice the last example here—we can convert to lowercase prior to the sort with a list comprehension, but the result does not contain the original list’s values as it does with the key argument. The latter is applied temporarily during the sort, instead of changing the values to be sorted altogether. As we move along, we’ll see contexts in which the sorted built-in can sometimes be more useful than the sort method.
Like strings, lists have other methods that perform other specialized operations. For instance, reverse reverses the list in-place, and the extend and pop methods insert multiple items at and delete an item from the end of the list, respectively. There is also a reversed built-in function that works much like sorted and returns a new result object, but it must be wrapped in a list call in both 2.X and 3.X here because its result is an iterator that produces results on demand (more on iterators later):
>>>L = [1, 2]>>>L.extend([3, 4, 5])# Add many items at end (like in-place +) >>>L[1, 2, 3, 4, 5] >>>L.pop()# Delete and return last item (by default: −1) 5 >>>L[1, 2, 3, 4] >>>L.reverse()# In-place reversal method >>>L[4, 3, 2, 1] >>>list(reversed(L))# Reversal built-in with a result (iterator) [1, 2, 3, 4]
Technically, the extend method always iterates through and adds each item in an iterable object, whereas append simply adds a single item as is without iterating through it—a distinction that will be more meaningful by Chapter 14. For now, it’s enough to know that extend adds many items, and append adds one. In some types of programs, the list pop method is often used in conjunction with append to implement a quick last-in-first-out (LIFO) stack structure. The end of the list serves as the top of the stack:
>>>L = []>>>L.append(1)# Push onto stack >>>L.append(2)>>>L[1, 2] >>>L.pop()# Pop off stack 2 >>>L[1]
The pop method also accepts an optional offset of the item to be deleted and returned (the default is the last item at offset −1). Other list methods remove an item by value (remove), insert an item at an offset (insert), count the number of occurrences (count), and search for an item’s offset (index—a search for the index of an item, not to be confused with indexing!):
>>>L = ['spam', 'eggs', 'ham']>>>L.index('eggs')# Indexofan object (search/find) 1 >>>L.insert(1, 'toast')# Insert at position >>>L['spam', 'toast', 'eggs', 'ham'] >>>L.remove('eggs')# Delete by value >>>L['spam', 'toast', 'ham'] >>>L.pop(1)# Delete by position 'toast' >>>L['spam', 'ham'] >>>L.count('spam')# Number of occurrences 1
Note that unlike other list methods, count and index do not change the list itself, but return information about its content. See other documentation sources or experiment with these calls interactively on your own to learn more about list methods.
Because lists are mutable, you can use the del statement to delete an item or section in place:
>>>L = ['spam', 'eggs', 'ham', 'toast']>>>del L[0]# Delete one item >>>L['eggs', 'ham', 'toast'] >>>del L[1:]# Delete an entire section >>>L# Same as L[1:] = [] ['eggs']
As we saw earlier, because slice assignment is a deletion plus an insertion, you can also delete a section of a list by assigning an empty list to a slice (L[i:j]=[]); Python deletes the slice named on the left, and then inserts nothing. Assigning an empty list to an index, on the other hand, just stores a reference to the empty list object in the specified slot, rather than deleting an item:
>>>L = ['Already', 'got', 'one']>>>L[1:] = []>>>L['Already'] >>>L[0] = []>>>L[[]]
Although all the operations just discussed are typical, there may be additional list methods and operations not illustrated here. The method set, for example, may change over time, and in fact has in Python 3.3—its new L.copy() method makes a top-level copy of the list, much like L[:] and list(L), but is symmetric with copy in sets and dictionaries. For a comprehensive and up-to-date list of type tools, you should always consult Python’s manuals, Python’s dir and help functions (which we first met in Chapter 4), or one of the reference texts mentioned in the preface.
And because it’s such a common hurdle, I’d also like to remind you again that all the in-place change operations discussed here work only for mutable objects: they won’t work on strings (or tuples, discussed in Chapter 9), no matter how hard you try. Mutability is an inherent property of each object type.
[19] This description requires elaboration when the value and the slice being assigned overlap: L[2:5]=L[3:6], for instance, works fine because the value to be inserted is fetched before the deletion happens on the left.
[20] Unlike + concatenation, append doesn’t have to generate new objects, so it’s usually faster than + too. You can also mimic append with the clever slice assignments of the prior section: L[len(L):]=[X] is like L.append(X), and L[:0]=[X] is like appending at the front of a list. Both delete an empty slice and insert X, changing L in place quickly, like append. Both are arguably more complex than list methods, though. For instance, L.insert(0, X) can also append an item to the front of a list, and seems noticeably more mnemonic; L.insert(len(L), X) inserts one object at the end too, but unless you like typing, you might as well use L.append(X)!