Timing Iterations and Pythons with timeit

For example, the timeit module’s repeat call returns a list giving the total time taken to run a test a number of times, for each of repeat runs—the min of this list yields the best time among the runs, and helps filter out system load fluctuations that can otherwise skew timing results artificially high.

The following shows this call in action, timing a list comprehension on two versions of CPython and the optimized PyPy implementation of Python described in Chapter 2 (it currently supports Python 2.7 code). The results here give the best total time in seconds among 5 runs that each execute the code string 1,000 times; the code string itself constructs a 1,000-item list of integers each time through (see Appendix B for the Windows launcher used for variety in the first two of these commands):

c:\code> py −3
Python 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:57:17) [MSC v.1600 64 bit...
>>> import timeit
>>> min(timeit.repeat(stmt="[x ** 2 for x in range(1000)]", number=1000, repeat=5))
0.5062382371756811

c:\code> py −2
Python 2.7.3 (default, Apr 10 2012, 23:24:47) [MSC v.1500 64 bit (AMD64)] on win32
>>> import timeit
>>> min(timeit.repeat(stmt="[x ** 2 for x in range(1000)]", number=1000, repeat=5))
0.0708020004193198

c:\code> c:\pypy\pypy-1.9\pypy.exe
Python 2.7.2 (341e1e3821ff, Jun 07 2012, 15:43:00)
[PyPy 1.9.0 with MSC v.1500 32 bit] on win32
>>>> import timeit
>>>> min(timeit.repeat(stmt="[x ** 2 for x in range(1000)]", number=1000, repeat=5))
0.0059330329674303905

You’ll notice that PyPy checks in at 10X faster than CPython 2.7 here, and a whopping 100X faster than CPython 3.3, despite the fact that PyPy is a potentially slower 32-bit build. This is a small artificial benchmark, of course, but seems arguably stunning nonetheless, and reflects a relative speed ranking that is generally supported by other tests run in this book (though as we’ll see, CPython still beats PyPy on some types of code).

This particular test measures the speed of both a list comprehension and integer math. The latter varies between lines: CPython 3.X has a single integer type, and CPython 2.X has both short and long integers. This may explain part of the size of the difference, but the results are valid nonetheless. Noninteger tests yield similar rankings (e.g., a floating-point test in the solutions to this part’s exercises), and integer math matters—the one and two order of magnitude (power of 10) speedups here will be realized by many real programs, because integers and iterations are ubiquitous in Python code.

These results also differ from the preceding section’s relative version speeds, where CPython 2.7 was slightly quicker than 3.3, and PyPy was 10X quicker overall, a figure affirmed by most other tests in this book too. Apart from the different type of code being timed here, the different coding structure inside timeit may have an effect too—for code strings like those tested here, timeit builds, compiles, and executes a function def statement string that embeds the test string, thereby avoiding a function call per inner loop. As we’ll see in the next section, though, this appears irrelevant from a relative-speed perspective.

The timeit module has reasonable defaults and can be also run as a script, either by explicit filename or automatically located on the module search path with Python’s –m flag (see Appendix A). All the following run Python (a.k.a. CPython) 3.3. In this mode timeit reports the average time for a single –n loop, in either microseconds (labeled “usec”), milliseconds (“msec”), or seconds (“sec”); to compare results here to the total time values reported by other tests, multiply by the number of loops run—500 usec here * 1,000 loops is 500 msec, or half a second in total time:

c:\code> C:\python33\Lib\timeit.py -n 1000 "[x ** 2 for x in range(1000)]"
1000 loops, best of 3: 506 usec per loop

c:\code> python -m timeit -n 1000 "[x ** 2 for x in range(1000)]"
1000 loops, best of 3: 504 usec per loop

c:\code> py −3 -m timeit -n 1000 -r 5 "[x ** 2 for x in range(1000)]"
1000 loops, best of 5: 505 usec per loop

As an example, we can use command lines to verify that choice of timer call doesn’t impact cross-version speed comparisons run in this chapter so far—3.3 uses its new calls by default, and that might matter if timer precision differs widely. To prove that this is irrelevant, the following uses the -c flag to force timeit to use time.clock in all versions, an option that 3.3’s manuals call deprecated, but required to even the score with prior versions (I’m setting my system path to include PyPy here for command brevity):

c:\code> set PATH=%PATH%;C:\pypy\pypy-1.9

c:\code> py −3 -m timeit -n 1000 -r 5 -c "[x ** 2 for x in range(1000)]"
1000 loops, best of 5: 502 usec per loop
c:\code> py −2 -m timeit -n 1000 -r 5 -c "[x ** 2 for x in range(1000)]"
1000 loops, best of 5: 70.6 usec per loop
c:\code> pypy -m timeit -n 1000 -r 5 -c  "[x ** 2 for x in range(1000)]"
1000 loops, best of 5: 5.44 usec per loop

C:\code> py −3 -m timeit -n 1000 -r 5 -c "[abs(x) for x in range(10000)]"
1000 loops, best of 5: 815 usec per loop
C:\code> py −2 -m timeit -n 1000 -r 5 -c "[abs(x) for x in range(10000)]"
1000 loops, best of 5: 700 usec per loop
C:\code> pypy -m timeit -n 1000 -r 5 -c  "[abs(x) for x in range(10000)]"
1000 loops, best of 5: 61.7 usec per loop

These results are essentially the same as those for earlier tests in this chapter on the same types of code. When applying x ** 2, CPython 2.7 and PyPy are again 10X and 100X faster than CPython 3.3, respectively, showing that timer choice isn’t a factor. For the abs(x) we timed under the homegrown timer earlier (timeseqs.py), these two Pythons are faster than 3.3 by a small constant and 10X just as before, implying that timeit’s different code structure doesn’t impact relative comparisons—the type of code being tested fully determines the size of speed differences.

Subtle point: notice that the results of the last three of these tests, which mimic tests run for the homegrown timer earlier, are basically the same as before, but seem to incur a small net overhead for range usage differences—it was a prebuilt list formerly, but here is either a 3.X generator or a 2.X list built anew on each inner total loop. In other words, we’re not timing the exact same thing, but the relative speeds of the Pythons tested are the same.

To time larger multiline sections of code in API call mode, use line breaks and tabs or spaces to satisfy Python’s syntax; code read from a source file already will. Because you pass Python string objects to a Python function in this mode, there are no shell considerations, though be careful to escape nested quotes if needed. The following, for instance, times Chapter 13 loop alternatives in Python 3.3; you can use the same pattern to time the file-line-reader alternatives in Chapter 14:

c:\code> py −3
>>> import timeit
>>> min(timeit.repeat(number=10000, repeat=3,
        stmt="L = [1, 2, 3, 4, 5]\nfor i in range(len(L)): L[i] += 1"))
0.01397292797131814

>>> min(timeit.repeat(number=10000, repeat=3,
        stmt="L = [1, 2, 3, 4, 5]\ni=0\nwhile i < len(L):\n\tL[i] += 1\n\ti += 1"))
0.015452276471516813

>>> min(timeit.repeat(number=10000, repeat=3,
        stmt="L = [1, 2, 3, 4, 5]\nM = [x + 1 for x in L]"))
0.009464995838568635

To run multiline statements like these in command-line mode, appease your shell by passing each statement line as a separate argument, with whitespace for indentation—timeit concatenates all the lines together with a newline character between them, and later reindents for its own statement nesting purposes. Leading spaces may work better for indentation than tabs in this mode, and be sure to quote the code arguments if required by your shell:

c:\code> py −3 -m timeit -n 1000 -r 3 "L = [1,2,3,4,5]" "i=0" "while i < len(L):"
 "    L[i] += 1" "    i += 1"
1000 loops, best of 3: 1.54 usec per loop

c:\code> py −3 -m timeit -n 1000 -r 3 "L = [1,2,3,4,5]" "M = [x + 1 for x in L]"
1000 loops, best of 3: 0.959 usec per loop

The timeit module also allows you to provide setup code that is run in the main statement’s scope, but whose time is not charged to the main statement’s total—potentially useful for initialization code you wish to exclude from total time, such as imports of required modules, test function definition, and test data creation. Because they’re run in the same scope, any names created by setup code are available to the main test statement; names defined in the interactive shell generally are not.

To specify setup code, use a –s in command-line mode (or many of these for multiline setups) and a setup argument string in API call mode. This can focus tests more sharply, as in the following, which splits list initialization off to a setup statement to time just iteration. As a rule of thumb, though, the more code you include in a test statement, the more applicable its results will generally be to realistic code:

c:\code> python -m timeit -n 1000 -r 3 "L = [1,2,3,4,5]" "M = [x + 1 for x in L]"
1000 loops, best of 3: 0.956 usec per loop

c:\code> python -m timeit -n 1000 -r 3 -s "L = [1,2,3,4,5]" "M = [x + 1 for x in L]"
1000 loops, best of 3: 0.775 usec per loop

Here’s a setup example in API call mode: I used the following type of code to time the sort-based option in Chapter 18’s minimum value example—ordered ranges sort much faster than random numbers, and are faster sorted than scanned linearly in the example’s code under 3.3 (adjacent strings are concatenated here):

>>> from timeit import repeat

>>> min(repeat(number=1000, repeat=3,
setup='from mins import min1, min2, min3\n'
      'vals=list(range(1000))',
stmt= 'min3(*vals)'))
0.0387865921275079

>>> min(repeat(number=1000, repeat=3,
setup='from mins import min1, min2, min3\n'
      'import random\nvals=[random.random() for i in range(1000)]',
stmt= 'min3(*vals)'))
0.275656482278373

With timeit, you can also ask for just total time, use the module’s class API, time callable objects instead of strings, accept automatic loop counts, and use class-based techniques and additional command-line switches and API argument options we don’t have space to show here—consult Python’s library manual for more details:

c:\code> py −3
>>> import timeit
>>> timeit.timeit(stmt='[x ** 2 for x in range(1000)]', number=1000)  # Total time
0.5238125259325834

>>> timeit.Timer(stmt='[x ** 2 for x in range(1000)]').timeit(1000)   # Class API
0.5282652329644009

>>> timeit.repeat(stmt='[x ** 2 for x in range(1000)]', number=1000, repeat=3)
[0.5299034147194845, 0.5082454007998365, 0.5095136232504416]

>>> def testcase():
        y = [x ** 2 for x in range(1000)]     # Callable objects or code strings

>>> min(timeit.repeat(stmt=testcase, number=1000, repeat=3))
0.5073828140463377

For example, the following tests in pybench_cases2.py measure the impact of charging other iteration operations with a function call, which improves map’s chances of winning the day per this chapter’s earlier note—map usually loses by its association with function calls in general:

# pybench_cases2.py

pythons += [
    (1, 'C:\python32\python'),
    (0, 'C:\pypy\pypy-2.0-beta1\pypy')]

stmts += [
# Use function calls: map wins
    (0, 0, "[ord(x) for x in 'spam' * 2500]"),
    (0, 0, "res=[]\nfor x in 'spam' * 2500: res.append(ord(x))"),
    (0, 0, "$listif3(map(ord, 'spam' * 2500))"),
    (0, 0, "list(ord(x) for x in 'spam' * 2500)"),
# Set and dicts
    (0, 0, "{x ** 2 for x in range(1000)}"),
    (0, 0, "s=set()\nfor x in range(1000): s.add(x ** 2)"),
    (0, 0, "{x: x ** 2 for x in range(1000)}"),
    (0, 0, "d={}\nfor x in range(1000): d[x] = x ** 2"),
# Pathological: 300k digits
    (1, 1, "len(str(2**1000000))")]  # Pypy loses on this today

Here is the script’s results on these statement tests on CPython 3.X, showing how map is quickest when function calls level the playing field (it lost earlier when the other tests ran an inline x ** 2):

c:\code> py −3 pybench_cases2.py
3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:57:17) [MSC v.1600 64 bit (AMD64)]
0.7237  ["[ord(x) for x in 'spam' * 2500]"]
1.3471  ["res=[]\nfor x in 'spam' * 2500: res.append(ord(x))"]
0.6160  ["list(map(ord, 'spam' * 2500))"]
1.1244  ["list(ord(x) for x in 'spam' * 2500)"]
0.5446  ['{x ** 2 for x in range(1000)}']
0.6053  ['s=set()\nfor x in range(1000): s.add(x ** 2)']
0.5278  ['{x: x ** 2 for x in range(1000)}']
0.5414  ['d={}\nfor x in range(1000): d[x] = x ** 2']
1.8933  ['len(str(2**1000000))']

As before, on these tests today 2.X clocks in faster than 3.X and PyPy is faster still on all of these tests but the last—which it loses by a full order of magnitude (10X), though it wins all the other tests here by the same degree. However, if you run file tests precoded in pybench_cases2.py you’ll see that PyPy also loses to CPython when reading files line by line, as for the following test tuple on the stmts list:

    (0, 0, "f=open('C:/Python33/Lib/pdb.py')\nfor line in f: x=line\nf.close()"),

This test opens and reads a 60K, 1,675-line text file line by line using file iterators. Its input loop presumably dominates overall test time. On this test, CPython 2.7 is twice as fast as 3.3, but PyPy is again an order of magnitude slower than CPython in general. You can find this case in the pybench_cases2 results files, or verify interactively or by command line (this is just what pybench does internally):

c:\code> py −3 -m timeit -n 1000 -r 5 "f=open('C:/Python33/Lib/pdb.py')"
 "for line in f: x=line" "f.close()"

>>> import timeit
>>> min(timeit.repeat(number=1000, repeat=5,
    stmt="f=open('C:/Python33/Lib/pdb.py')\nfor line in f: x=line\nf.close()"))

For another example that measures both list comprehensions and PyPy’s current file speed, see the file listcomp-speed.txt in the book examples package; it uses direct PyPy command lines to run code from Chapter 14 with similar results: PyPy’s line input is slower today by roughly a factor of 10.

I’ll omit other Pythons’ output here both for space and because these findings could very well change by the time you read these words. As usual, different types of code can exhibit different types of performance. While PyPy may optimize much algorithmic code, it may or may not optimize yours. You can find additional results in the book’s examples package, but you may be better served by running these tests on your own to verify these findings today or observe their possibly different results in the future.

As suggested earlier, map also wins for added user-defined functions—the following tests prove the earlier note’s claim that map wins the race in CPython if any function must be applied by its alternatives:

stmts = [
    (0, 0, "def f(x): return x\n[f(x) for x in 'spam' * 2500]"),
    (0, 0, "def f(x): return x\nres=[]\nfor x in 'spam' * 2500: res.append(f(x))"),
    (0, 0, "def f(x): return x\n$listif3(map(f, 'spam' * 2500))"),
    (0, 0, "def f(x): return x\nlist(f(x) for x in 'spam' * 2500)")]

c:\code> py −3 pybench_cases2.py
3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:57:17) [MSC v.1600 64 bit (AMD64)]
1.5400  ["def f(x): return x\n[f(x) for x in 'spam' * 2500]"]
2.0506  ["def f(x): return x\nres=[]\nfor x in 'spam' * 2500: res.append(f(x))"]
1.2489  ["def f(x): return x\nlist(map(f, 'spam' * 2500))"]
1.6526  ["def f(x): return x\nlist(f(x) for x in 'spam' * 2500)"]

Compare this with the preceding section’s ord tests; though user-defined functions may be slower than built-ins, the larger speed hit today seems to be functions in general, whether they are built-in or not. Notice that the total time here includes the cost of making a helper function, though only one for every 10,000 inner loop repetitions—a negligible factor per both common sense and additional tests run.

For perspective, let’s see how this section’s timeit-based results compare to the homegrown-based timer results of the prior section, by running the file timeseqs3.py in this book’s examples package—it uses the homegrown timer but performs the same x ** 2 operation and uses the same repetition counts as pybench_cases.py:

c:\code> py −3 timeseqs3.py
3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:57:17) [MSC v.1600 64 bit (AMD64)]
forLoop  : 0.55022 => [0...998001]
listComp : 0.48787 => [0...998001]
mapCall  : 0.59499 => [0...998001]
genExpr  : 0.52773 => [0...998001]
genFunc  : 0.52603 => [0...998001]

c:\code> py −3 pybench_cases.py
3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:57:17) [MSC v.1600 64 bit (AMD64)]
0.5015  ['[x ** 2 for x in range(1000)]']
0.5657  ['res=[]\nfor x in range(1000): res.append(x ** 2)']
0.6025  ['list(map(lambda x: x ** 2, range(1000)))']
0.5404  ['list(x ** 2 for x in range(1000))']
0.8711  ["s = 'spam' * 2500\nx = [s[i] for i in range(10000)]"]
2.8009  ["s = '?'\nfor i in range(10000): s += '?'"]

The homegrown timer results are very similar to the pybench-based results of this section that use timeit, though it’s not entirely apples-to-apples—the homegrown timer-based timeseqs3.py incurs a function call per its middle totals loop and a slight overhead in best of logic of the timer itself, but also uses a prebuilt list instead of a 3.X range generator in its inner loop, which seems to make it slightly net faster on comparable tests (and I’d call this example a “sanity check,” but I’m not sure the term applies in benchmarking!).

Like most software, this section’s program is open-ended and could be expanded arbitrarily. As one example, the files pybench2.py and pybench2_cases.py in the book’s examples package add support for timeit’s setup statement option described earlier, in both API call and command-line modes.

This feature was omitted initially for brevity, and frankly, because my tests didn’t seem to require it—timing more code gives a more complete picture when comparing Pythons, and setup actions cost the same when timing alternatives on a single Python. Even so, it’s sometimes useful to provide setup code that is run once in the tested code’s scope, but whose time is not charged to the statement’s total—a module import, object initialization, or helper function definition, for example.

I won’t list these two files in whole, but here are their important varying bits as an example of software evolution at work—as for the test statement, the setup code statement is passed as is in API call mode, but is split and space-indented in command-line mode and passed with one -s argument per line (“$listif3” isn’t used because setup code is not timed):

# pybench2.py
...
def runner(stmts, pythons=None, tracecmd=False):
    for (number, repeat, setup, stmt) in stmts:
        if not pythons:
            ...
            best = min(timeit.repeat(
                              setup=setup, stmt=stmt, number=number, repeat=repeat))
        else:
            setup = setup.replace('\t', ' ' * 4)
            setup = ' '.join('-s "%s"' % line for line in setup.split('\n'))
            ...
            for (ispy3, python) in pythons:
                ...
                cmd = '%s -m timeit -n %s -r %s %s %s' %
                              (python, number, repeat, setup, args)

# pybench2_cases.py
import pybench2, sys
...
stmts = [                                                     # (num,rpt,setup,stmt)
    (0, 0, "", "[x ** 2 for x in range(1000)]"),
    (0, 0, "", "res=[]\nfor x in range(1000): res.append(x ** 2)"),

    (0, 0, "def f(x):\n\treturn x",
           "[f(x) for x in 'spam' * 2500]"),
    (0, 0, "def f(x):\n\treturn x",
           "res=[]\nfor x in 'spam' * 2500:\n\tres.append(f(x))"),

    (0, 0, "L = [1, 2, 3, 4, 5]", "for i in range(len(L)): L[i] += 1"),
    (0, 0, "L = [1, 2, 3, 4, 5]", "i=0\nwhile i < len(L):\n\tL[i] += 1\n\ti += 1")]
...
pybench2.runner(stmts, pythons, tracecmd)

Run this script with the –a and –t command-line flags to see how command lines are constructed for setup code. For instance, the following test specification tuple generates the command line that follows it for 3.3—not nice to look at, perhaps, but sufficient to pass lines from Windows to timeit, to be concatenated with line breaks between and inserted into a generated timing function with appropriate reindentation:

    (0, 0, "def f(x):\n\treturn x",
           "res=[]\nfor x in 'spam' * 2500:\n\tres.append(f(x))")

C:\python33\python -m timeit -n 1000 -r 5 -s "def f(x):" -s "    return x" "res=[]"
 "for x in 'spam' * 2500:" "    res.append(f(x))"

In API call mode, code strings are passed unchanged, because there’s no need to placate a shell, and embedded tabs and end-of-line characters suffice. Experiment on your own to uncover more about Python code alternatives’ speed. You may eventually run into shell limitations for larger sections of code in command-line mode, but both our homegrown timer and pybench’s timeit-based API call mode support more arbitrary code. Benchmarks can be great sport, but we’ll have to leave future improvements as suggested exercises.