Core Language Summary

Built-in types like strings, lists, and dictionaries make it easy to write simple programs fast.

For more demanding tasks, you can extend Python by writing your own functions, modules, and classes.

Although we don’t cover this topic in this book, Python can also be extended with modules written in an external language like C or C++.

PyDoc’s help function and HTML interfaces were introduced in Chapter 15. PyDoc provides a documentation system for your modules and objects, integrates with Python’s docstrings syntax, and is a standard part of the Python system. See Chapter 15 and Chapter 4 for more documentation source hints.

Because Python is such a dynamic language, some programming errors are not reported until your program runs (even syntax errors are not caught until a file is run or imported). This isn’t a big drawback—as with most languages, it just means that you have to test your Python code before shipping it. At worst, with Python you essentially trade a compile phase for an initial testing phase. Furthermore, Python’s dynamic nature, automatic error messages, and exception model make it easier and quicker to find and fix errors than it is in some other languages. Unlike C, for example, Python does not crash completely on errors.

Still, tools can help here too. The PyChecker and PyLint systems provide support for catching common errors ahead of time, before your script runs. They serve similar roles to the lint program in C development. Some Python developers run their code through PyChecker prior to testing or delivery, to catch any lurking potential problems. In fact, it’s not a bad idea to try this when you’re first starting out—some of these tools’ warnings may help you learn to spot and avoid common Python mistakes. PyChecker and PyLint are third-party open source packages, available at the PyPI website or your friendly neighborhood web search engine. They may appear in IDE GUIs as well.

In Chapter 25, we learned how to add self-test code to a Python file by using the __name__ == '__main__' trick at the bottom of the file—a simple unit-testing protocol. For more advanced testing purposes, Python comes with two testing support tools. The first, PyUnit (called unittest in the library manual), provides an object-oriented class framework for specifying and customizing test cases and expected results. It mimics the JUnit framework for Java. This is a sophisticated class-based unit testing system; see the Python library manual for details.

The doctest standard library module provides a second and simpler approach to regression testing, based upon Python’s docstrings feature. Roughly, to use doctest, you cut and paste a log of an interactive testing session into the docstrings of your source files. doctest then extracts your docstrings, parses out the test cases and results, and reruns the tests to verify the expected results. doctest’s operation can be tailored in a variety of ways; see the library manual for more details.

We discussed IDEs for Python in Chapter 3. IDEs such as IDLE provide a graphical environment for editing, running, debugging, and browsing your Python programs. Some advanced IDEs—such as Eclipse, Komodo, NetBeans, and others listed in Chapter 3—may support additional development tasks, including source control integration, code refactoring, project management tools, and more. See Chapter 3, the text editors page at http://www.python.org, and your favorite web search engine for more on available IDEs and GUI builders for Python.

Because Python is so high-level and dynamic, intuitions about performance gleaned from experience with other languages usually don’t apply to Python code. To truly isolate performance bottlenecks in your code, you need to add timing logic with clock tools in the time or timeit modules, or run your code under the profile module. We saw an example of the timing modules at work when comparing the speed of iteration tools and Pythons in Chapter 21.

Profiling is usually your first optimization step—code for clarity, then profile to isolate bottlenecks, and then time alternative codings of the slow parts of your program. For the second of these steps, profile is a standard library module that implements a source code profiler for Python. It runs a string of code you provide (e.g., a script file import, or a call to a function) and then, by default, prints a report to the standard output stream that gives performance statistics—number of calls to each function, time spent in each function, and more.

The profile module can be run as a script or imported, and it may be customized in various ways; for example, it can save run statistics to a file to be analyzed later with the pstats module. To profile interactively, import the profile module and call profile.run('code'), passing in the code you wish to profile as a string (e.g., a call to a function, an import of a file, or code read from a file). To profile from a system shell command line, use a command of the form python -m profile main.py args (see Appendix A for more on this format). Also see Python’s standard library manuals for other profiling options; the cProfile module, for example, has identical interfaces to profile but runs with less overhead, so it may be better suited to profiling long-running programs.

We also discussed debugging options in Chapter 3 (see its sidebar Debugging Python Code). As a review, most development IDEs for Python support GUI-based debugging, and the Python standard library also includes a source code debugger module called pdb. This module provides a command-line interface and works much like common C language debuggers (e.g., dbx, gdb).

Much like the profiler, the pdb debugger can be run either interactively or from a command line and can be imported and called from a Python program. To use it interactively, import the module, start running code by calling a pdb function (e.g., pdb.run('main()')), and then type debugging commands from pdb’s interactive prompt. To launch pdb from a system shell command line, use a command of the form python -m pdb main.py args. pdb also includes a useful postmortem analysis call, pdb.pm(), which starts the debugger after an exception has been encountered, possibly in conjunction with Python’s -i flag. See Appendix A for more on these tools.

Because IDEs such as IDLE also include point-and-click debugging interfaces, pdb isn’t as critical a tool today, except when a GUI isn’t available or when more control is desired. See Chapter 3 for tips on using IDLE’s debugging GUI interfaces. Really, neither pdb nor IDEs seem to be used much in practice—as noted in Chapter 3, most programmers either insert print statements or simply read Python’s error messages: perhaps not the most high-tech of approaches, but the practical tends to win the day in the Python world!

In Chapter 2, we introduced common tools for packaging Python programs. py2exe, PyInstaller, and others listed in that chapter can package byte code and the Python Virtual Machine into “frozen binary” standalone executables, which don’t require that Python be installed on the target machine and hide your system’s code. In addition, we learned in Chapter 2 that Python programs may be shipped in their source (.py) or byte code (.pyc) forms, and that import hooks support special packaging techniques such as automatic extraction of .zip files and byte code encryption.

We also briefly met the standard library’s distutils modules, which provide packaging options for Python modules and packages, and C-coded extensions; see the Python manuals for more details. The emerging Python “eggs” third-party packaging system provides another alternative that also accounts for dependencies; search the Web for more details.

When speed counts, there are a handful of options for optimizing your programs. The PyPy system described in Chapter 2 provides a just-in-time compiler for translating Python byte code to binary machine code, and Shed Skin offers a Python-to-C++ translator. You may also occasionally see .pyo optimized byte code files, generated and run with the -O Python command-line flag discussed in Chapter 22 and Chapter 34, and to be deployed in Chapter 39; because this provides a very modest performance boost, however, it is not commonly used except to remove debugging code.

As a last resort, you can also move parts of your program to a compiled language such as C to boost performance. See the book Programming Python and the Python standard manuals for more on C extensions. In general, Python’s speed tends to also improve over time, so upgrading to later releases may improve speed too—once you verify that they are faster for your code, that is (though largely repaired since, Python 3.0’s initial release was up to 1000X slower than 2.X on some IO operations!).

We’ve met a variety of core language features in this text that will also tend to become more useful once you start coding larger projects. These include module packages (Chapter 24), class-based exceptions (Chapter 34), class pseudoprivate attributes (Chapter 31), documentation strings (Chapter 15), module path configuration files (Chapter 22), hiding names from from * with __all__ lists and _X-style names (Chapter 25), adding self-test code with the __name__ == '__main__' trick (Chapter 25), using common design rules for functions and modules (Chapter 17, Chapter 19, and Chapter 25), using object-oriented design patterns (Chapter 31 and others), and so on.