Module Gotchas

If you have two modules of the same name, you may only be able to import one of them—by default, the one whose directory is leftmost in the sys.path module search path will always be chosen. This isn’t an issue if the module you prefer is in your top-level script’s directory; since that is always first in the module path, its contents will be located first automatically. For cross-directory imports, however, the linear nature of the module search path means that same-named files can clash.

To fix, either avoid same-named files or use the package imports feature of Chapter 24. If you need to get to both same-named files, structure your source files in subdirectories, such that package import directory names make the module references unique. As long as the enclosing package directory names are unique, you’ll be able to access either or both of the same-named modules.

Note that this issue can also crop up if you accidentally use a name for a module of your own that happens to be the same as a standard library module you need—your local module in the program’s home directory (or another directory early in the module path) can hide and replace the library module.

To fix, either avoid using the same name as another module you need or store your modules in a package directory and use Python 3.X’s package-relative import model, available in 2.X as an option. In this model, normal imports skip the package directory (so you’ll get the library’s version), but special dotted import statements can still select the local version of the module if needed.

As we’ve seen, when a module is first imported (or reloaded), Python executes its statements one by one, from the top of the file to the bottom. This has a few subtle implications regarding forward references that are worth underscoring here:

Generally, forward references are only a concern in top-level module code that executes immediately; functions can reference names arbitrarily. Here’s a file that illustrates forward reference dos and don’ts:

func1()                           # Error: "func1" not yet assigned

def func1():
    print(func2())                # OK: "func2" looked up later

func1()                           # Error: "func2" not yet assigned

def func2():
    return "Hello"

func1()                           # OK: "func1" and "func2" assigned

When this file is imported (or run as a standalone program), Python executes its statements from top to bottom. The first call to func1 fails because the func1 def hasn’t run yet. The call to func2 inside func1 works as long as func2’s def has been reached by the time func1 is called—and it hasn’t when the second top-level func1 call is run. The last call to func1 at the bottom of the file works because func1 and func2 have both been assigned.

Mixing defs with top-level code is not only difficult to read, it’s also dependent on statement ordering. As a rule of thumb, if you need to mix immediate code with defs, put your defs at the top of the file and your top-level code at the bottom. That way, your functions are guaranteed to be defined and assigned by the time Python runs the code that uses them.

Although it’s commonly used, the from statement is the source of a variety of potential gotchas in Python. As we’ve learned, the from statement is really an assignment to names in the importer’s scope—a name-copy operation, not a name aliasing. The implications of this are the same as for all assignments in Python, but they’re subtle, especially given that the code that shares the objects lives in different files. For instance, suppose we define the following module, nested1.py:

# nested1.py
X = 99
def printer(): print(X)

If we import its two names using from in another module, nested2.py, we get copies of those names, not links to them. Changing a name in the importer resets only the binding of the local version of that name, not the name in nested1.py:

# nested2.py
from nested1 import X, printer    # Copy names out
X = 88                            # Changes my "X" only!
printer()                         # nested1's X is still 99

% python nested2.py
99

If we use import to get the whole module and then assign to a qualified name, however, we change the name in nested1.py. Attribute qualification directs Python to a name in the module object, rather than a name in the importer, nested3.py:

# nested3.py
import nested1                    # Get module as a whole
nested1.X = 88                    # OK: change nested1's X
nested1.printer()

% python nested3.py
88

I mentioned this earlier but saved the details for here. Because you don’t list the variables you want when using the from module import * statement form, it can accidentally overwrite names you’re already using in your scope. Worse, it can make it difficult to determine where a variable comes from. This is especially true if the from * form is used on more than one imported file.

For example, if you use from * on three modules in the following, you’ll have no way of knowing what a raw function call really means, short of searching all three external module files—all of which may be in other directories:

>>> from module1 import *          # Bad: may overwrite my names silently
>>> from module2 import *          # Worse: no way to tell what we get!
>>> from module3 import *
>>> . . .

>>> func()                         # Huh???

The solution again is not to do this: try to explicitly list the attributes you want in your from statements, and restrict the from * form to at most one imported module per file. That way, any undefined names must by deduction be in the module named in the single from *. You can avoid the issue altogether if you always use import instead of from, but that advice is too harsh; like much else in programming, from is a convenient tool if used wisely. Even this example isn’t an absolute evil—it’s OK for a program to use this technique to collect names in a single space for convenience, as long as it’s well known.

Here’s another from-related gotcha: as discussed previously, because from copies (assigns) names when run, there’s no link back to the modules where the names came from. Names imported with from simply become references to objects, which happen to have been referenced by the same names in the importee when the from ran.

Because of this behavior, reloading the importee has no effect on clients that import its names using from. That is, the client’s names will still reference the original objects fetched with from, even if the names in the original module are later reset:

from module import X          # X may not reflect any module reloads!
 . . .
from imp import reload
reload(module)                # Changes module, but not my names
X                             # Still references old object

To make reloads more effective, use import and name qualification instead of from. Because qualifications always go back to the module, they will find the new bindings of module names after reloading has updated the module’s content in place:

import module                 # Get module, not names
 . . .
from imp import reload
reload(module)                # Changes module in place
module.X                      # Get current X: reflects module reloads

As a related consequence, our transitive reloader earlier in this chapter doesn’t apply to names fetched with from, only import; again, if you’re going to use reloads, you’re probably better off with import.

In fact, the prior gotcha is even more subtle than it appears. Chapter 3 warned that it’s usually better not to launch programs with imports and reloads because of the complexities involved. Things get even worse when from is brought into the mix. Python beginners most often stumble onto its issues in scenarios like this—imagine that after opening a module file in a text edit window, you launch an interactive session to load and test your module with from:

from module import function
function(1, 2, 3)

Finding a bug, you jump back to the edit window, make a change, and try to reload the module this way:

from imp import reload
reload(module)

This doesn’t work, because the from statement assigned only the name function, not module. To refer to the module in a reload, you have to first bind its name with an import statement at least once:

from imp import reload
import module
reload(module)
function(1, 2, 3)

However, this doesn’t quite work either—reload updates the module object in place, but as discussed in the preceding section, names like function that were copied out of the module in the past still refer to the old objects; in this instance, function is still the original version of the function. To really get the new function, you must refer to it as module.function after the reload, or rerun the from:

from imp import reload
import module
reload(module)
from module import function        # Or give up and use module.function()
function(1, 2, 3)

Now, the new version of the function will finally run, but it seems an awful lot of work to get there.

As you can see, there are problems inherent in using reload with from: not only do you have to remember to reload after imports, but you also have to remember to rerun your from statements after reloads. This is complex enough to trip up even an expert once in a while. In fact, the situation has gotten even worse in Python 3.X, because you must also remember to import reload itself!

The short story is that you should not expect reload and from to play together nicely. Again, the best policy is not to combine them at all—use reload with import, or launch your programs other ways, as suggested in Chapter 3: using the Run→Run Module menu option in IDLE, file icon clicks, system command lines, or the exec built-in function.

I saved the most bizarre (and, thankfully, obscure) gotcha for last. Because imports execute a file’s statements from top to bottom, you need to be careful when using modules that import each other. This is often called recursive imports, but the recursion doesn’t really occur (in fact, circular may be a better term here)—such imports won’t get stuck in infinite importing loops. Still, because the statements in a module may not all have been run when it imports another module, some of its names may not yet exist.

If you use import to fetch the module as a whole, this probably doesn’t matter; the module’s names won’t be accessed until you later use qualification to fetch their values, and by that time the module is likely complete. But if you use from to fetch specific names, you must bear in mind that you will only have access to names in that module that have already been assigned when a recursive import is kicked off.

For instance, consider the following modules, recur1 and recur2. recur1 assigns a name X, and then imports recur2 before assigning the name Y. At this point, recur2 can fetch recur1 as a whole with an import—it already exists in Python’s internal modules table, which makes it importable, and also prevents the imports from looping. But if recur2 uses from, it will be able to see only the name X; the name Y, which is assigned below the import in recur1, doesn’t yet exist, so you get an error:

# recur1.py
X = 1
import recur2                             # Run recur2 now if it doesn't exist
Y = 2

# recur2.py
from recur1 import X                      # OK: "X" already assigned
from recur1 import Y                      # Error: "Y" not yet assigned

C:\code> py −3
>>> import recur1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".\recur1.py", line 2, in <module>
    import recur2
  File ".\recur2.py", line 2, in <module>
    from recur1 import Y
ImportError: cannot import name Y

Python avoids rerunning recur1’s statements when they are imported recursively from recur2 (otherwise the imports would send the script into an infinite loop that might require a Ctrl-C solution or worse), but recur1’s namespace is incomplete when it’s imported by recur2.

The solution? Don’t use from in recursive imports (no, really!). Python won’t get stuck in a cycle if you do, but your programs will once again be dependent on the order of the statements in the modules. In fact, there are two ways out of this gotcha:

There is additional perspective on this issue in the exercises at the end of this chapter—which we’ve officially reached.