Other Numeric Types

The last point merits elaboration. As previewed briefly when we explored comparisons, floating-point math is less than exact because of the limited space used to store values. For example, the following should yield zero, but it does not. The result is close to zero, but there are not enough bits to be precise here:

>>> 0.1 + 0.1 + 0.1 - 0.3                         # Python 3.3
5.551115123125783e-17

On Pythons prior to 3.1 and 2.7, printing the result to produce the user-friendly display format doesn’t completely help either, because the hardware related to floating-point math is inherently limited in terms of accuracy (a.k.a. precision). The following in 3.3 gives the same result as the previous output:

>>> print(0.1 + 0.1 + 0.1 - 0.3)                  # Pythons < 2.7, 3.1
5.55111512313e-17

However, with decimals, the result can be dead-on:

>>> from decimal import Decimal
>>> Decimal('0.1') + Decimal('0.1') + Decimal('0.1') - Decimal('0.3')
Decimal('0.0')

As shown here, we can make decimal objects by calling the Decimal constructor function in the decimal module and passing in strings that have the desired number of decimal digits for the resulting object (using the str function to convert floating-point values to strings if needed). When decimals of different precision are mixed in expressions, Python converts up to the largest number of decimal digits automatically:

>>> Decimal('0.1') + Decimal('0.10') + Decimal('0.10') - Decimal('0.30')
Decimal('0.00')

In Pythons 2.7, 3.1, and later, it’s also possible to create a decimal object from a floating-point object, with a call of the form decimal.Decimal.from_float(1.25), and recent Pythons allow floating-point numbers to be used directly. The conversion is exact but can sometimes yield a large default number of digits, unless they are fixed per the next section:

>>> Decimal(0.1) + Decimal(0.1) + Decimal(0.1) - Decimal(0.3)
Decimal('2.775557561565156540423631668E-17')

In Python 3.3 and later, the decimal module was also optimized to improve its performance radically: the reported speedup for the new version is 10X to 100X, depending on the type of program benchmarked.

Other tools in the decimal module can be used to set the precision of all decimal numbers, arrange error handling, and more. For instance, a context object in this module allows for specifying precision (number of decimal digits) and rounding modes (down, ceiling, etc.). The precision is applied globally for all decimals created in the calling thread:

>>> import decimal
>>> decimal.Decimal(1) / decimal.Decimal(7)                     # Default: 28 digits
Decimal('0.1428571428571428571428571429')

>>> decimal.getcontext().prec = 4                               # Fixed precision
>>> decimal.Decimal(1) / decimal.Decimal(7)
Decimal('0.1429')

>>> Decimal(0.1) + Decimal(0.1) + Decimal(0.1) - Decimal(0.3)   # Closer to 0
Decimal('1.110E-17')

This is especially useful for monetary applications, where cents are represented as two decimal digits. Decimals are essentially an alternative to manual rounding and string formatting in this context:

>>> 1999 + 1.33      # This has more digits in memory than displayed in 3.3
2000.33
>>>
>>> decimal.getcontext().prec = 2
>>> pay = decimal.Decimal(str(1999 + 1.33))
>>> pay
Decimal('2000.33')

In Python 2.6 and 3.0 and later, it’s also possible to reset precision temporarily by using the with context manager statement. The precision is reset to its original value on statement exit; in a new Python 3.3 session (per Chapter 3 the “...” here is Python’s interactive prompt for continuation lines in some interfaces and requires manual indentation; IDLE omits this prompt and indents for you):

C:\code> C:\Python33\python
>>> import decimal
>>> decimal.Decimal('1.00') / decimal.Decimal('3.00')
Decimal('0.3333333333333333333333333333')
>>>
>>> with decimal.localcontext() as ctx:
...     ctx.prec = 2
...     decimal.Decimal('1.00') / decimal.Decimal('3.00')
...
Decimal('0.33')
>>>
>>> decimal.Decimal('1.00') / decimal.Decimal('3.00')
Decimal('0.3333333333333333333333333333')

Though useful, this statement requires much more background knowledge than you’ve obtained at this point; watch for coverage of the with statement in Chapter 34.

Because use of the decimal type is still relatively rare in practice, I’ll defer to Python’s standard library manuals and interactive help for more details. And because decimals address some of the same floating-point accuracy issues as the fraction type, let’s move on to the next section to see how the two compare.

Fraction is a functional cousin to the Decimal fixed-precision type described in the prior section, as both can be used to address the floating-point type’s numerical inaccuracies. It’s also used in similar ways—like Decimal, Fraction resides in a module; import its constructor and pass in a numerator and a denominator to make one (among other schemes). The following interaction shows how:

>>> from fractions import Fraction
>>> x = Fraction(1, 3)                    # Numerator, denominator
>>> y = Fraction(4, 6)                    # Simplified to 2, 3 by gcd

>>> x
Fraction(1, 3)
>>> y
Fraction(2, 3)
>>> print(y)
2/3

Once created, Fractions can be used in mathematical expressions as usual:

>>> x + y
Fraction(1, 1)
>>> x − y                           # Results are exact: numerator, denominator
Fraction(−1, 3)
>>> x * y
Fraction(2, 9)

Fraction objects can also be created from floating-point number strings, much like decimals:

>>> Fraction('.25')
Fraction(1, 4)
>>> Fraction('1.25')
Fraction(5, 4)
>>>
>>> Fraction('.25') + Fraction('1.25')
Fraction(3, 2)

Notice that this is different from floating-point-type math, which is constrained by the underlying limitations of floating-point hardware. To compare, here are the same operations run with floating-point objects, and notes on their limited accuracy—they may display fewer digits in recent Pythons than they used to, but they still aren’t exact values in memory:

>>> a = 1 / 3.0                     # Only as accurate as floating-point hardware
>>> b = 4 / 6.0                     # Can lose precision over many calculations
>>> a
0.3333333333333333
>>> b
0.6666666666666666

>>> a + b
1.0
>>> a - b
-0.3333333333333333
>>> a * b
0.2222222222222222

This floating-point limitation is especially apparent for values that cannot be represented accurately given their limited number of bits in memory. Both Fraction and Decimal provide ways to get exact results, albeit at the cost of some speed and code verbosity. For instance, in the following example (repeated from the prior section), floating-point numbers do not accurately give the zero answer expected, but both of the other types do:

>>> 0.1 + 0.1 + 0.1 - 0.3           # This should be zero (close, but not exact)
5.551115123125783e-17

>>> from fractions import Fraction
>>> Fraction(1, 10) + Fraction(1, 10) + Fraction(1, 10) - Fraction(3, 10)
Fraction(0, 1)

>>> from decimal import Decimal
>>> Decimal('0.1') + Decimal('0.1') + Decimal('0.1') - Decimal('0.3')
Decimal('0.0')

Moreover, fractions and decimals both allow more intuitive and accurate results than floating points sometimes can, in different ways—by using rational representation and by limiting precision:

>>> 1 / 3                           # Use a ".0" in Python 2.X for true "/"
0.3333333333333333

>>> Fraction(1, 3)                  # Numeric accuracy, two ways
Fraction(1, 3)

>>> import decimal
>>> decimal.getcontext().prec = 2
>>> Decimal(1) / Decimal(3)
Decimal('0.33')

In fact, fractions both retain accuracy and automatically simplify results. Continuing the preceding interaction:

>>> (1 / 3) + (6 / 12)              # Use a ".0" in Python 2.X for true "/"
0.8333333333333333

>>> Fraction(6, 12)                 # Automatically simplified
Fraction(1, 2)

>>> Fraction(1, 3) + Fraction(6, 12)
Fraction(5, 6)

>>> decimal.Decimal(str(1/3)) + decimal.Decimal(str(6/12))
Decimal('0.83')

>>> 1000.0 / 1234567890
8.100000073710001e-07
>>> Fraction(1000, 1234567890)      # Substantially simpler!
Fraction(100, 123456789)

To support fraction conversions, floating-point objects now have a method that yields their numerator and denominator ratio, fractions have a from_float method, and float accepts a Fraction as an argument. Trace through the following interaction to see how this pans out (the * in the second test is special syntax that expands a tuple into individual arguments; more on this when we study function argument passing in Chapter 18):

>>> (2.5).as_integer_ratio()               # float object method
(5, 2)

>>> f = 2.5
>>> z = Fraction(*f.as_integer_ratio())    # Convert float -> fraction: two args
>>> z                                      # Same as Fraction(5, 2)
Fraction(5, 2)

>>> x                                      # x from prior interaction
Fraction(1, 3)
>>> x + z
Fraction(17, 6)                            # 5/2 + 1/3 = 15/6 + 2/6

>>> float(x)                               # Convert fraction -> float
0.3333333333333333
>>> float(z)
2.5
>>> float(x + z)
2.8333333333333335
>>> 17 / 6
2.8333333333333335

>>> Fraction.from_float(1.75)              # Convert float -> fraction: other way
Fraction(7, 4)
>>> Fraction(*(1.75).as_integer_ratio())
Fraction(7, 4)

Finally, some type mixing is allowed in expressions, though Fraction must sometimes be manually propagated to retain accuracy. Study the following interaction to see how this works:

>>> x
Fraction(1, 3)
>>> x + 2                                  # Fraction + int -> Fraction
Fraction(7, 3)
>>> x + 2.0                                # Fraction + float -> float
2.3333333333333335
>>> x + (1./3)                             # Fraction + float -> float
0.6666666666666666
>>> x + (4./3)
1.6666666666666665
>>> x + Fraction(4, 3)                     # Fraction + Fraction -> Fraction
Fraction(5, 3)

Caveat: although you can convert from floating point to fraction, in some cases there is an unavoidable precision loss when you do so, because the number is inaccurate in its original floating-point form. When needed, you can simplify such results by limiting the maximum denominator value:

>>> 4.0 / 3
1.3333333333333333
>>> (4.0 / 3).as_integer_ratio()                # Precision loss from float
(6004799503160661, 4503599627370496)

>>> x
Fraction(1, 3)
>>> a = x + Fraction(*(4.0 / 3).as_integer_ratio())
>>> a
Fraction(22517998136852479, 13510798882111488)

>>> 22517998136852479 / 13510798882111488.      # 5 / 3 (or close to it!)
1.6666666666666667

>>> a.limit_denominator(10)                     # Simplify to closest fraction
Fraction(5, 3)

For more details on the Fraction type, experiment further on your own and consult the Python 2.6, 2.7, and 3.X library manuals and other documentation.

There are a few ways to make sets today, depending on which Python you use. Since this book covers all, let’s begin with the case for 2.6 and earlier, which also is available (and sometimes still required) in later Pythons; we’ll refine this for 2.7 and 3.X extensions in a moment. To make a set object, pass in a sequence or other iterable object to the built-in set function:

>>> x = set('abcde')
>>> y = set('bdxyz')

You get back a set object, which contains all the items in the object passed in (notice that sets do not have a positional ordering, and so are not sequences—their order is arbitrary and may vary per Python release):

>>> x
set(['a', 'c', 'b', 'e', 'd'])                    # Pythons <= 2.6 display format

Sets made this way support the common mathematical set operations with expression operators. Note that we can’t perform the following operations on plain sequences like strings, lists, and tuples—we must create sets from them by passing them to set in order to apply these tools:

>>> x − y                                         # Difference
set(['a', 'c', 'e'])

>>> x | y                                         # Union
set(['a', 'c', 'b', 'e', 'd', 'y', 'x', 'z'])

>>> x & y                                         # Intersection
set(['b', 'd'])

>>> x ^ y                                         # Symmetric difference (XOR)
set(['a', 'c', 'e', 'y', 'x', 'z'])

>>> x > y, x < y                                  # Superset, subset
(False, False)

The notable exception to this rule is the in set membership test—this expression is also defined to work on all other collection types, where it also performs membership (or a search, if you prefer to think in procedural terms). Hence, we do not need to convert things like strings and lists to sets to run this test:

>>> 'e' in x                                      # Membership (sets)
True

>>> 'e' in 'Camelot', 22 in [11, 22, 33]          # But works on other types too
(True, True)

In addition to expressions, the set object provides methods that correspond to these operations and more, and that support set changes—the set add method inserts one item, update is an in-place union, and remove deletes an item by value (run a dir call on any set instance or the set type name to see all the available methods). Assuming x and y are still as they were in the prior interaction:

>>> z = x.intersection(y)                         # Same as x & y
>>> z
set(['b', 'd'])
>>> z.add('SPAM')                                 # Insert one item
>>> z
set(['b', 'd', 'SPAM'])
>>> z.update(set(['X', 'Y']))                     # Merge: in-place union
>>> z
set(['Y', 'X', 'b', 'd', 'SPAM'])
>>> z.remove('b')                                 # Delete one item
>>> z
set(['Y', 'X', 'd', 'SPAM'])

As iterable containers, sets can also be used in operations such as len, for loops, and list comprehensions. Because they are unordered, though, they don’t support sequence operations like indexing and slicing:

>>> for item in set('abc'): print(item * 3)

aaa
ccc
bbb

Finally, although the set expressions shown earlier generally require two sets, their method-based counterparts can often work with any iterable type as well:

>>> S = set([1, 2, 3])

>>> S | set([3, 4])          # Expressions require both to be sets
set([1, 2, 3, 4])
>>> S | [3, 4]
TypeError: unsupported operand type(s) for |: 'set' and 'list'

>>> S.union([3, 4])          # But their methods allow any iterable
set([1, 2, 3, 4])
>>> S.intersection((1, 3, 5))
set([1, 3])
>>> S.issubset(range(-5, 5))
True

For more details on set operations, see Python’s library reference manual or a reference book. Although set operations can be coded manually in Python with other types, like lists and dictionaries (and often were in the past), Python’s built-in sets use efficient algorithms and implementation techniques to provide quick and standard operation.

If you think sets are “cool,” they eventually became noticeably cooler, with new syntax for set literals and comprehensions initially added in the Python 3.X line only, but back-ported to Python 2.7 by popular demand. In these Pythons we can still use the set built-in to make set objects, but also a new set literal form, using the curly braces formerly reserved for dictionaries. In 3.X and 2.7, the following are equivalent:

set([1, 2, 3, 4])                # Built-in call (all)
{1, 2, 3, 4}                     # Newer set literals (2.7, 3.X)

This syntax makes sense, given that sets are essentially like valueless dictionaries—because a set’s items are unordered, unique, and immutable, the items behave much like a dictionary’s keys. This operational similarity is even more striking given that dictionary key lists in 3.X are view objects, which support set-like behavior such as intersections and unions (see Chapter 8 for more on dictionary view objects).

Regardless of how a set is made, 3.X displays it using the new literal format. Python 2.7 accepts the new literal syntax, but still displays sets using the 2.6 display form of the prior section. In all Pythons, the set built-in is still required to create empty sets and to build sets from existing iterable objects (short of using set comprehensions, discussed later in this chapter), but the new literal is convenient for initializing sets of known structure.

Here’s what sets look like in 3.X; it’s the same in 2.7, except that set results display with 2.X’s set([...]) notation, and item order may vary per version (which by definition is irrelevant in sets anyhow):

C:\code> c:\python33\python
>>> set([1, 2, 3, 4])            # Built-in: same as in 2.6
{1, 2, 3, 4}
>>> set('spam')                  # Add all items in an iterable
{'s', 'a', 'p', 'm'}

>>> {1, 2, 3, 4}                 # Set literals: new in 3.X (and 2.7)
{1, 2, 3, 4}
>>> S = {'s', 'p', 'a', 'm'}
>>> S
{'s', 'a', 'p', 'm'}

>>> S.add('alot')                # Methods work as before
>>> S
{'s', 'a', 'p', 'alot', 'm'}

All the set processing operations discussed in the prior section work the same in 3.X, but the result sets print differently:

>>> S1 = {1, 2, 3, 4}
>>> S1 & {1, 3}                  # Intersection
{1, 3}
>>> {1, 5, 3, 6} | S1            # Union
{1, 2, 3, 4, 5, 6}
>>> S1 - {1, 3, 4}               # Difference
{2}
>>> S1 > {1, 3}                  # Superset
True

Note that {} is still a dictionary in all Pythons. Empty sets must be created with the set built-in, and print the same way:

>>> S1 - {1, 2, 3, 4}            # Empty sets print differently
set()
>>> type({})                     # Because {} is an empty dictionary
<class 'dict'>

>>> S = set()                    # Initialize an empty set
>>> S.add(1.23)
>>> S
{1.23}

As in Python 2.6 and earlier, sets created with 3.X/2.7 literals support the same methods, some of which allow general iterable operands that expressions do not:

>>> {1, 2, 3} | {3, 4}
{1, 2, 3, 4}
>>> {1, 2, 3} | [3, 4]
TypeError: unsupported operand type(s) for |: 'set' and 'list'

>>> {1, 2, 3}.union([3, 4])
{1, 2, 3, 4}
>>> {1, 2, 3}.union({3, 4})
{1, 2, 3, 4}
>>> {1, 2, 3}.union(set([3, 4]))
{1, 2, 3, 4}

>>> {1, 2, 3}.intersection((1, 3, 5))
{1, 3}
>>> {1, 2, 3}.issubset(range(-5, 5))
True

Sets are powerful and flexible objects, but they do have one constraint in both 3.X and 2.X that you should keep in mind—largely because of their implementation, sets can only contain immutable (a.k.a. “hashable”) object types. Hence, lists and dictionaries cannot be embedded in sets, but tuples can if you need to store compound values. Tuples compare by their full values when used in set operations:

>>> S
{1.23}
>>> S.add([1, 2, 3])                   # Only immutable objects work in a set
TypeError: unhashable type: 'list'
>>> S.add({'a':1})
TypeError: unhashable type: 'dict'
>>> S.add((1, 2, 3))
>>> S                                  # No list or dict, but tuple OK
{1.23, (1, 2, 3)}

>>> S | {(4, 5, 6), (1, 2, 3)}         # Union: same as S.union(...)
{1.23, (4, 5, 6), (1, 2, 3)}
>>> (1, 2, 3) in S                     # Membership: by complete values
True
>>> (1, 4, 3) in S
False

Tuples in a set, for instance, might be used to represent dates, records, IP addresses, and so on (more on tuples later in this part of the book). Sets may also contain modules, type objects, and more. Sets themselves are mutable too, and so cannot be nested in other sets directly; if you need to store a set inside another set, the frozenset built-in call works just like set but creates an immutable set that cannot change and thus can be embedded in other sets.

In addition to literals, Python 3.X grew a set comprehension construct that was back-ported for use to Python 2.7 too. Like the 3.X set literal, 2.7 accepts its syntax, but displays its results in 2.X set notation. The set comprehension expression is similar in form to the list comprehension we previewed in Chapter 4, but is coded in curly braces instead of square brackets and run to make a set instead of a list. Set comprehensions run a loop and collect the result of an expression on each iteration; a loop variable gives access to the current iteration value for use in the collection expression. The result is a new set you create by running the code, with all the normal set behavior. Here is a set comprehension in 3.3 (again, result display and order differs in 2.7):

>>> {x ** 2 for x in [1, 2, 3, 4]}         # 3.X/2.7 set comprehension
{16, 1, 4, 9}

In this expression, the loop is coded on the right, and the collection expression is coded on the left (x ** 2). As for list comprehensions, we get back pretty much what this expression says: “Give me a new set containing X squared, for every X in a list.” Comprehensions can also iterate across other kinds of objects, such as strings (the first of the following examples illustrates the comprehension-based way to make a set from an existing iterable):

>>> {x for x in 'spam'}                    # Same as: set('spam')
{'m', 's', 'p', 'a'}

>>> {c * 4 for c in 'spam'}                # Set of collected expression results
{'pppp', 'aaaa', 'ssss', 'mmmm'}
>>> {c * 4 for c in 'spamham'}
{'pppp', 'aaaa', 'hhhh', 'ssss', 'mmmm'}

>>> S = {c * 4 for c in 'spam'}
>>> S | {'mmmm', 'xxxx'}
{'pppp', 'xxxx', 'mmmm', 'aaaa', 'ssss'}
>>> S & {'mmmm', 'xxxx'}
{'mmmm'}

Because the rest of the comprehensions story relies upon underlying concepts we’re not yet prepared to address, we’ll postpone further details until later in this book. In Chapter 8, we’ll meet a first cousin in 3.X and 2.7, the dictionary comprehension, and I’ll have much more to say about all comprehensions—list, set, dictionary, and generator—later on, especially in Chapter 14 and Chapter 20. As we’ll learn there, all comprehensions support additional syntax not shown here, including nested loops and if tests, which can be challenging to understand until you’ve had a chance to study larger statements.

Set operations have a variety of common uses, some more practical than mathematical. For example, because items are stored only once in a set, sets can be used to filter duplicates out of other collections, though items may be reordered in the process because sets are unordered in general. Simply convert the collection to a set, and then convert it back again (sets work in the list call here because they are iterable, another technical artifact that we’ll unearth later):

>>> L = [1, 2, 1, 3, 2, 4, 5]
>>> set(L)
{1, 2, 3, 4, 5}
>>> L = list(set(L))                                  # Remove duplicates
>>> L
[1, 2, 3, 4, 5]

>>> list(set(['yy', 'cc', 'aa', 'xx', 'dd', 'aa']))   # But order may change
['cc', 'xx', 'yy', 'dd', 'aa']

Sets can be used to isolate differences in lists, strings, and other iterable objects too—simply convert to sets and take the difference—though again the unordered nature of sets means that the results may not match that of the originals. The last two of the following compare attribute lists of string object types in 3.X (results vary in 2.7):

>>> set([1, 3, 5, 7]) - set([1, 2, 4, 5, 6])          # Find list differences
{3, 7}
>>> set('abcdefg') - set('abdghij')                   # Find string differences
{'c', 'e', 'f'}
>>> set('spam') - set(['h', 'a', 'm'])                # Find differences, mixed
{'p', 's'}

>>> set(dir(bytes)) - set(dir(bytearray))             # In bytes but not bytearray
{'__getnewargs__'}
>>> set(dir(bytearray)) - set(dir(bytes))
{'append', 'copy', '__alloc__', '__imul__', 'remove', 'pop', 'insert', ...more...]

You can also use sets to perform order-neutral equality tests by converting to a set before the test, because order doesn’t matter in a set. More formally, two sets are equal if and only if every element of each set is contained in the other—that is, each is a subset of the other, regardless of order. For instance, you might use this to compare the outputs of programs that should work the same but may generate results in different order. Sorting before testing has the same effect for equality, but sets don’t rely on an expensive sort, and sorts order their results to support additional magnitude tests that sets do not (greater, less, and so on):

>>> L1, L2 = [1, 3, 5, 2, 4], [2, 5, 3, 4, 1]
>>> L1 == L2                                          # Order matters in sequences
False
>>> set(L1) == set(L2)                                # Order-neutral equality
True
>>> sorted(L1) == sorted(L2)                          # Similar but results ordered
True
>>> 'spam' == 'asmp', set('spam') == set('asmp'), sorted('spam') == sorted('asmp')
(False, True, True)

Sets can also be used to keep track of where you’ve already been when traversing a graph or other cyclic structure. For example, the transitive module reloader and inheritance tree lister examples we’ll study in Chapter 25 and Chapter 31, respectively, must keep track of items visited to avoid loops, as Chapter 19 discusses in the abstract. Using a list in this context is inefficient because searches require linear scans. Although recording states visited as keys in a dictionary is efficient, sets offer an alternative that’s essentially equivalent (and may be more or less intuitive, depending on whom you ask).

Finally, sets are also convenient when you’re dealing with large data sets (database query results, for example)—the intersection of two sets contains objects common to both categories, and the union contains all items in either set. To illustrate, here’s a somewhat more realistic example of set operations at work, applied to lists of people in a hypothetical company, using 3.X/2.7 set literals and 3.X result displays (use set in 2.6 and earlier):

>>> engineers = {'bob', 'sue', 'ann', 'vic'}
>>> managers  = {'tom', 'sue'}

>>> 'bob' in engineers                   # Is bob an engineer?
True

>>> engineers & managers                 # Who is both engineer and manager?
{'sue'}

>>> engineers | managers                 # All people in either category
{'bob', 'tom', 'sue', 'vic', 'ann'}

>>> engineers - managers                 # Engineers who are not managers
{'vic', 'ann', 'bob'}

>>> managers - engineers                 # Managers who are not engineers
{'tom'}

>>> engineers > managers                 # Are all managers engineers? (superset)
False

>>> {'bob', 'sue'} < engineers           # Are both engineers? (subset)
True

>>> (managers | engineers) > managers    # All people is a superset of managers
True

>>> managers ^ engineers                 # Who is in one but not both?
{'tom', 'vic', 'ann', 'bob'}

>>> (managers | engineers) - (managers ^ engineers)     # Intersection!
{'sue'}

You can find more details on set operations in the Python library manual and some mathematical and relational database theory texts. Also stay tuned for Chapter 8’s revival of some of the set operations we’ve seen here, in the context of dictionary view objects in Python 3.X.