Now that you’ve learned all about package and package-relative imports, I need to explain that there’s a new option that modifies some of the ideas we just covered. At least abstractly, as of release 3.3 Python has four import models. From original to newest:
import mod, from mod import attrThe original model: imports of files and their contents, relative to the sys.path module search path
import dir1.dir2.mod, from dir1.mod import attrImports that give directory path extensions relative to the sys.path module search path, where each package is contained in a single directory and has an initialization file, in Python 2.X and 3.X
from . import mod (relative), import mod (absolute)The model used for intrapackage imports of the prior section, with its relative or absolute lookup schemes for dotted and nondotted imports, available but differing in Python 2.X and 3.X
import splitdir.modThe new namespace package model that we’ll survey here, which allows packages to span multiple directories, and requires no initialization file, introduced in Python 3.3
The first two of these are self-contained, but the third tightens up the search order and extends syntax for intrapackage imports, and the fourth upends some of the core notions and requirements of the prior package model. In fact, Python 3.3 (and later) now has two flavors of packages:
The original model, now known as regular packages
The alternative model, known as namespace packages
This is similar in spirit to the “classic” and “new style” class model dichotomy we’ll meet in the next part of this book, though the new is more an addition to the old here. The original and new package models are not mutually exclusive, and can be used simultaneously in the same program. In fact, the new namespace package model works as something of a fallback option, recognized only if normal modules and regular packages of the same name are not present on the module search path.
The rationale for namespace packages is rooted in package installation goals that may seem obscure unless you are responsible for such tasks, and is better addressed by this feature’s PEP document. In short, though, they resolve a potential for collision of multiple __init__.py files when package parts are merged, by removing this file completely. Moreover, by providing standard support for packages that can be split across multiple directories and located in multiple sys.path entries, namespace packages both enhance install flexibility and provide a common mechanism to replace the multiple incompatible solutions that have arisen to address this goal.
Though too early to judge their uptake, average Python users may find namespace packages to be a useful and alternative extension to the regular package model—one that does not require initialization files, and allows any directory of code to be used as an importable package. To see why, let’s move on to the details.
A namespace package is not fundamentally different from a regular package; it is just a different way of creating packages. Moreover, they are still relative to sys.path at the top level: the leftmost component of a dotted namespace package path must still be located in an entry on the normal module search path.
In terms of physical structure, though, the two can differ substantially. Regular packages still must have an __init__.py file that is run automatically, and reside in a single directory as before. By contrast, new-style namespace packages cannot contain an __init__.py, and may span multiple directories that are collected at import time. In fact, none of the directories that make up a namespace package can have an __init__.py, but the content nested within each of them is treated as a single package.
To truly understand namespace packages, we have to look under the hood to see how the import operation works in 3.3. During imports, Python still iterates over each directory in the module search path, sys.path, just as in 3.2 and earlier. In 3.3, though, while looking for an imported module or package named spam, for each directory in the module search path, Python tests for a wider variety of matching criteria, in the following order:
If directory\spam\__init__.py is found, a regular package is imported and returned.
If directory\spam.{py, pyc, or other module extension} is found, a simple module is imported and returned.
If directory\spam is found and is a directory, it is recorded and the scan continues with the next directory in the search path.
If none of the above was found, the scan continues with the next directory in the search path.
If the search path scan completes without returning a module or package by steps 1 or 2, and at least one directory was recorded by step 3, then a namespace package is created.
The creation of the namespace package happens immediately, and is not deferred until a sublevel import occurs. The new namespace package has a __path__ attribute set to an iterable of the directory path strings that were found and recorded during the scan by step 3, but does not have a __file__.
The __path__ attribute is then used in later, deeper accesses to search all package components—each recorded entry on a namespace package’s __path__ is searched whenever further nested items are requested, much like the sole directory of a regular package.
Viewed another way, the __path__ attribute of a namespace package serves the same role for lower-level components that sys.path does at the top for the leftmost component of package import paths; it becomes the “parent path” for accessing lower items using the same four-step procedure just sketched.
The net result is that a namespace package is a sort of virtual concatenation of directories located via multiple sys.path entries. Once a namespace package is created, though, there is no functional difference between it and a regular package; it supports everything we’ve learned for regular packages, including package-relative import syntax.
As one consequence of this new import procedure, as of Python 3.3 packages no longer require __init__.py files—when a single-directory package does not have this file, it will be treated as a single-directory namespace package, and no warning will be issued. This is a major relaxation of prior rules, but a commonly requested change; many packages require no initialization code, and it seemed extraneous to have to create an empty initialization file in such cases. This is finally no longer required as of 3.3.
At the same time, the original regular package model is still fully supported, and automatically runs code in __init__.py as before as an initialization hook. Moreover, when it’s known that a package will never be a portion of a split namespace package, there is a performance advantage to coding it as a regular package with an __init__.py. Creation and loading of a regular package occurs immediately when it is located along the path. With namespace packages, all entries in the path must be scanned before the package is created. More formally, regular packages stop the prior section’s algorithm at step 1; namespace packages do not.
Per this change’s PEP, there is no plan to remove support of regular packages—at least, that’s the story today; change is always a possibility in open source projects (indeed, the prior edition quoted plans on string formatting and relative imports in 2.X that were later abandoned), so as usual, be sure to watch for future developments on this front. Given the performance advantage and auto-initialization code of regular packages, though, it seems unlikely that they would be removed altogether.
To see how namespace packages work, consider the following two modules and nested directory structure—with two subdirectories named sub located in different parent directories, dir1 and dir2:
C:\code\ns\dir1\sub\mod1.py C:\code\ns\dir2\sub\mod2.py
If we add both dir1 and dir2 to the module search path, sub becomes a namespace package spanning both, with the two module files available under that name even though they live in separate physical directories. Here’s the files’ contents and the required path settings on Windows: there are no __init__.py files here—in fact there cannot be in namespace packages, as this is their chief physical differentiation:
c:\code>mkdir ns\dir1\sub# Two dirs of same name in different dirs c:\code>mkdir ns\dir2\sub# And similar outside Windows c:\code>type ns\dir1\sub\mod1.py# Module files in different directories print(r'dir1\sub\mod1') c:\code>type ns\dir2\sub\mod2.pyprint(r'dir2\sub\mod2') c:\code>set PYTHONPATH=C:\code\ns\dir1;C:\code\ns\dir2
Now, when imported directly in 3.3 and later, the namespace package is the virtual concatenation of its individual directory components, and allows further nested parts to be accessed through its single, composite name with normal imports:
c:\code>C:\Python33\python>>>import sub>>>sub# Namespace packages: nested search paths <module 'sub' (namespace)> >>>sub.__path___NamespacePath(['C:\\code\\ns\\dir1\\sub', 'C:\\code\\ns\\dir2\\sub']) >>>from sub import mod1dir1\sub\mod1 >>>import sub.mod2# Content from two different directories dir2\sub\mod2 >>>mod1<module 'sub.mod1' from 'C:\\code\\ns\\dir1\\sub\\mod1.py'> >>>sub.mod2<module 'sub.mod2' from 'C:\\code\\ns\\dir2\\sub\\mod2.py'>
This is also true if we import through the namespace package name immediately—because the namespace package is made when first reached, the timing of path extensions is irrelevant:
c:\code>C:\Python33\python>>>import sub.mod1dir1\sub\mod1 >>>import sub.mod2# One package spanning two directories dir2\sub\mod2 >>>sub.mod1<module 'sub.mod1' from 'C:\\code\\ns\\dir1\\sub\\mod1.py'> >>>sub.mod2<module 'sub.mod2' from 'C:\\code\\ns\\dir2\\sub\\mod2.py'> >>>sub<module 'sub' (namespace)> >>>sub.__path___NamespacePath(['C:\\code\\ns\\dir1\\sub', 'C:\\code\\ns\\dir2\\sub'])
Interestingly, relative imports work in namespace packages too—in the following, the relative import statement references a file in the package, even though the referenced file resides in a different directory:
c:\code>type ns\dir1\sub\mod1.pyfrom . import mod2 # And "from . import string" still fails print(r'dir1\sub\mod1') c:\code>C:\Python33\python>>>import sub.mod1# Relative import of mod2 in another dir dir2\sub\mod2 dir1\sub\mod1 >>>import sub.mod2# Already imported module not rerun >>>sub.mod2<module 'sub.mod2' from 'C:\\code\\ns\\dir2\\sub\\mod2.py'>
As you can see, namespace packages are like ordinary single-directory packages in every way, except for having a split physical storage—which is why single directory namespaces packages without __init__.py files are exactly like regular packages, but with no initialization logic to be run.
Namespace packages even support arbitrary nesting—once a package namespace package is created, it serves essentially the same role at its level that sys.path does at the top, becoming the “parent path” for lower levels. Continuing the prior section’s example:
c:\code>mkdir ns\dir2\sub\lower# Further nested components c:\code>type ns\dir2\sub\lower\mod3.pyprint(r'dir2\sub\lower\mod3') c:\code>C:\Python33\python>>>import sub.lower.mod3# Namespace pkg nested in namespace pkg dir2\sub\lower\mod3 c:\code>C:\Python33\python>>>import sub# Same effect if accessed incrementally >>>import sub.mod2dir2\sub\mod2 >>>import sub.lower.mod3dir2\sub\lower\mod3 >>>sub.lower# A single-directory namespace pkg <module 'sub.lower' (namespace)> >>>sub.lower.__path___NamespacePath(['C:\\code\\ns\\dir2\\sub\\lower'])
In the preceding, sub is a namespace package split across two directories, and sub.lower is a single-directory namespace package nested within the portion of sub physically located in dir2. sub.lower is also the namespace package equivalent of a regular package with no __init__.py.
This nesting behavior holds true whether the lower component is a module, regular package, or another namespace package—by serving as new import search paths, namespace packages allow all three to be nested within them freely:
c:\code>mkdir ns\dir1\sub\pkgC:\code>type ns\dir1\sub\pkg\__init__.pyprint(r'dir1\sub\pkg\__init__.py') c:\code>C:\Python33\python>>>import sub.mod2# Nested module dir2\sub\mod2 >>>import sub.pkg# Nested regular package dir1\sub\pkg\__init__.py >>>import sub.lower.mod3# Nested namespace package dir2\sub\lower\mod3 >>>sub# Modules, packages,and namespaces <module 'sub' (namespace)> >>>sub.mod2<module 'sub.mod2' from 'C:\\code\\ns\\dir2\\sub\\mod2.py'> >>>sub.pkg<module 'sub.pkg' from 'C:\\code\\ns\\dir1\\sub\\pkg\\__init__.py'> >>>sub.lower<module 'sub.lower' (namespace)> >>>sub.lower.mod3<module 'sub.lower.mod3' from 'C:\\code\\ns\\dir2\\sub\\lower\\mod3.py'>
Trace through this example’s files and directories for more insight. As you can see, namespace packages integrate seamlessly into the former import models, and extend it with new functionality.
As explained earlier, part of the purpose of __init___.py files in regular packages is to declare the directory as a package—it tells Python to use the directory, rather than skipping ahead to a possible file of the same name later on the path. This avoids inadvertently choosing a noncode subdirectory that accidentally appears early on the path, over a desired module of the same name.
Because namespace packages do not require these special files, they would seem to invalidate this safeguard. This isn’t the case, though—because the namespace algorithm outlined earlier continues scanning the path after a namespace directory has been found, files later on the path still have priority over earlier directories with no __init__.py. For example, consider the following directories and modules:
c:\code>mkdir ns2c:\code>mkdir ns3c:\code>mkdir ns3\dirc:\code>notepad ns3\dir\ns2.pyc:\code>type ns3\dir\ns2.pyprint(r'ns3\dir\ns2.py!')
The ns2 directory here cannot be imported in Python 3.2 and earlier—it’s not a regular package, as it lacks an __init__.py initialization file. This directory can be imported under 3.3, though—it’s a namespace package directory in the current working directory, which is always the first item on the sys.path module search path irrespective of PYTHONPATH settings:
c:\code>set PYTHONPATH=c:\code>py −3.2>>>import ns2ImportError: No module named ns2 c:\code>py −3.3>>>import ns2>>>ns2# A single-directory namespace package in CWD <module 'ns2' (namespace)> >>>ns2.__path___NamespacePath(['.\\ns2'])
But watch what happens when the directory containing a file of the same name as a namespace directory is added later on the search path, via PYTHONPATH settings—the file is used instead, because Python keeps searching later path entries after a namespace package directory is found. It stops searching only when a module or regular package is located, or the path has been completely scanned. Namespace packages are returned only if nothing else was found along the way:
c:\code>set PYTHONPATH=C:\code\ns3\dirc:\code>py −3.3>>>import ns2# Use later module file, not same-named directory! ns3\dir\ns2.py! >>>ns2<module 'ns2' from 'C:\\code\\ns3\\dir\\ns2.py'> >>>import sys>>>sys.path[:2]# First '' means current working directory, CWD ['', 'C:\\code\\ns3\\dir']
In fact, setting the path to include a module works the same as it does in earlier Pythons, even if a same-named namespace directory appears earlier on the path; namespace packages are used in 3.3 only in cases that would be errors in earlier Pythons:
c:\code>py −3.2>>>import ns2ns3\dir\ns2.py! >>>ns2<module 'ns2' from 'C:\code\ns3\dir\ns2.py'>
This is also why none of the directories in a namespace package is allowed to have a __init__.py file: as soon as the import algorithm finds one that does, it returns a regular package immediately, and abandons the path search and the namespace package. Put more formally, the import algorithm chooses a namespace package only at the end of the path scan, and stops at steps 1 or 2 if either a regular package or module file is found sooner.
The net effect is that both module files and regular packages anywhere on the module search path have precedence over namespace package directories. In the following, for example, a namespace package called sub exists as the concatenation of same-named directories under dir1 and dir2 on the path:
c:\code>mkdir ns4\dir1\subc:\code>mkdir ns4\dir2\subc:\code>set PYTHONPATH=c:\code\ns4\dir1;c:\code\ns4\dir2c:\code>py −3>>>import sub>>>sub<module 'sub' (namespace)> >>>sub.__path___NamespacePath(['c:\\code\\ns4\\dir1\\sub', 'c:\\code\\ns4\\dir2\\sub'])
Much like a module file, though, a regular package added in the rightmost path entry takes priority over same-named namespace package directories too—the import path scan starts recording a namespace package tentatively in dir1 as before, but abandons it when the regular package is detected in dir2:
c:\code>notepad ns4\dir2\sub\__init__.pyc:\code>py −3>>>import sub# Use later reg. package, not same-named directory! >>>sub<module 'sub' from 'c:\\code\\ns4\\dir2\\sub\\__init__.py'>
Though a useful extension, because namespace packages are available only to readers using Python 3.3 (and later) I’m going to defer to Python’s manuals for more details on the subject. See especially this change’s PEP document for this change’s rationale, additional details, and more comprehensive examples.