Byte Code Files: __pycache__ in Python 3.2+

As mentioned briefly, the way that Python stores files to retain the byte code that results from compiling your source has changed in Python 3.2 and later. First of all, if Python cannot write a file to save this on your computer for any reason, your program still runs fine—Python simply creates and uses the byte code in memory and discards it on exit. To speed startups, though, it will try to save byte code in a file in order to skip the compile step next time around. The way it does this varies per Python version:

In Python 3.1 and earlier (including all of Python 2.X)

Byte code is stored in files in the same directory as the corresponding source files, normally with the filename extension .pyc (e.g., module.pyc). Byte code files are also stamped internally with the version of Python that created them (known as a “magic” field to developers) so Python knows to recompile when this differs in the version of Python running your program. For instance, if you upgrade to a new Python whose byte code differs, all your byte code files will be recompiled automatically due to a version number mismatch, even if you haven’t changed your source code.

In Python 3.2 and later

Byte code is instead stored in files in a subdirectory named __pycache__, which Python creates if needed, and which is located in the directory containing the corresponding source files. This helps avoid clutter in your source directories by segregating the byte code files in their own directory. In addition, although byte code files still get the .pyc extension as before, they are given more descriptive names that include text identifying the version of Python that created them (e.g., module.cpython-32.pyc). This avoids contention and recompiles: because each version of Python installed can have its own uniquely named version of byte code files in the __pycache__ subdirectory, running under a given version doesn’t overwrite the byte code of another, and doesn’t require recompiles. Technically, byte code filenames also include the name of the Python that created them, so CPython, Jython, and other implementations mentioned in the preface and Chapter 2 can coexist on the same machine without stepping on each other’s work (once they support this model).

In both models, Python always recreates the byte code file if you’ve changed the source code file since the last compile, but version differences are handled differently—by magic numbers and replacement prior to 3.2, and by filenames that allow for multiple copies in 3.2 and later.

Byte Code File Models in Action

The following is a quick example of these two models in action under 2.X and 3.3. I’ve omitted much of the text displayed by the dir directory listing on Windows here to save space, and the script used here isn’t listed because it is not relevant to this discussion (it’s from Chapter 2, and simply prints two values). Prior to 3.2, byte code files show up alongside their source files after being created by import operations:

c:\code\py2x> dir
10/31/2012  10:58 AM                39 script0.py

c:\code\py2x> C:\python27\python
>>> import script0
hello world
1267650600228229401496703205376
>>> ^Z

c:\code\py2x> dir
10/31/2012  10:58 AM                39 script0.py
10/31/2012  11:00 AM               154 script0.pyc

However, in 3.2 and later byte code files are saved in the __pycache__ subdirectory and include versions and Python implementation details in their names to avoid clutter and contention among the Pythons on your computer:

c:\code\py2x> cd ..\py3x
c:\code\py3x> dir
10/31/2012  10:58 AM                39 script0.py

c:\code\py3x> C:\python33\python
>>> import script0
hello world
1267650600228229401496703205376
>>> ^Z

c:\code\py3x> dir
10/31/2012  10:58 AM                39 script0.py
10/31/2012  11:00 AM    <DIR>          __pycache__

c:\code\py3x> dir __pycache__
10/31/2012  11:00 AM               184 script0.cpython-33.pyc

Crucially, under the model used in 3.2 and later, importing the same file with a different Python creates a different byte code file, instead of overwriting the single file as done by the pre-3.2 model—in the newer model, each Python version and implementation has its own byte code files, ready to be loaded on the next program run (earlier Pythons will happily continue using their scheme on the same machine):

c:\code\py3x> C:\python32\python
>>> import script0
hello world
1267650600228229401496703205376
>>> ^Z

c:\code\py3x> dir __pycache__
10/31/2012  12:28 PM               178 script0.cpython-32.pyc
10/31/2012  11:00 AM               184 script0.cpython-33.pyc

Python 3.2’s newer byte code file model is probably superior, as it avoids recompiles when there is more than one Python on your machine—a common case in today’s mixed 2.X/3.X world. On the other hand, it is not without potential incompatibilities in programs that rely on the prior file and directory structure. This may be a compatibility issue in some tools programs, for instance, though most well-behaved tools should work as before. See Python 3.2’s “What’s New?” document for details on potential impacts.

Also keep in mind that this process is completely automatic—it’s a side effect of running programs—and most programmers probably won’t care about or even notice the difference, apart from faster startups due to fewer recompiles.