This section develops a module tool that ties together and applies some earlier topics, and serves as a larger case study to close out this chapter and part. We studied module reloads in Chapter 23, as a way to pick up changes in code without stopping and restarting a program. When you reload a module, though, Python reloads only that particular module’s file; it doesn’t automatically reload modules that the file being reloaded happens to import.
For example, if you reload some module A, and A imports modules B and C, the reload applies only to A, not to B and C. The statements inside A that import B and C are rerun during the reload, but they just fetch the already loaded B and C module objects (assuming they’ve been imported before). In actual yet abstract code, here’s the file A.py:
# A.py import B # Not reloaded when A is! import C # Just an import of an already loaded module: no-ops %python>>> . . . >>>from imp import reload>>>reload(A)
By default, this means that you cannot depend on reloads to pick up changes in all the modules in your program transitively—instead, you must use multiple reload calls to update the subcomponents independently. This can require substantial work for large systems you’re testing interactively. You can design your systems to reload their subcomponents automatically by adding reload calls in parent modules like A, but this complicates the modules’ code.
A better approach is to write a general tool to do transitive reloads automatically by scanning modules’ __dict__ namespace attributes and checking each item’s type to find nested modules to reload. Such a utility function could call itself recursively to navigate arbitrarily shaped and deep import dependency chains. Module __dict__ attributes were introduced in Chapter 23 and employed earlier in this chapter, and the type call was presented in Chapter 9; we just need to combine the two tools.
The module reloadall.py listed next defines a reload_all function that automatically reloads a module, every module that the module imports, and so on, all the way to the bottom of each import chain. It uses a dictionary to keep track of already reloaded modules, recursion to walk the import chains, and the standard library’s types module, which simply predefines type results for built-in types. The visited dictionary technique works to avoid cycles here when imports are recursive or redundant, because module objects are immutable and so can be dictionary keys; as we learned in Chapter 5 and Chapter 8, a set would offer similar functionality if we use visited.add(module) to insert:
#!python """ reloadall.py: transitively reload nested modules (2.X + 3.X). Call reload_all with one or more imported module module objects. """ import types from imp import reload # from required in 3.X def status(module): print('reloading ' + module.__name__) def tryreload(module): try: reload(module) # 3.3 (only?) fails on some except: print('FAILED: %s' % module) def transitive_reload(module, visited): if not module in visited: # Trap cycles, duplicates status(module) # Reload this module tryreload(module) # And visit children visited[module] = True for attrobj in module.__dict__.values(): # For all attrs if type(attrobj) == types.ModuleType: # Recur if module transitive_reload(attrobj, visited) def reload_all(*args): visited = {} # Main entry point for arg in args: # For all passed in if type(arg) == types.ModuleType: transitive_reload(arg, visited) def tester(reloader, modname): # Self-test code import importlib, sys # Import on tests only if len(sys.argv) > 1: modname = sys.argv[1] # command line (or passed) module = importlib.import_module(modname) # Import by name string reloader(module) # Test passed-in reloader if __name__ == '__main__': tester(reload_all, 'reloadall') # Test: reload myself?
Besides namespace dictionaries, this script makes use of other tools we’ve studied here: it includes a __name__ test to launch self-test code when run as a top-level script only, and its tester function uses sys.argv to inspect command-line arguments and importlib to import a module by name string passed in as a function or command-line argument. One curious bit: notice how this code must wrap the basic reload call in a try statement to catch exceptions—in Python 3.3, reloads sometimes fail due to a rewrite of the import machinery. The try was previewed in Chapter 10, and is covered in full in Part VII.
Now, to leverage this utility for normal use, import its reload_all function and pass it an already loaded module object—just as you would for the built-in reload function. When the file runs standalone, its self-test code calls reload_all automatically, reloading its own module by default if no command-line arguments are used. In this mode, the module must import itself because its own name is not defined in the file without an import. This code works in both 3.X and 2.X because we’ve used + and % instead of a comma in the prints, though the set of modules used and thus reloaded may vary across lines:
C:\code>c:\Python33\python reloadall.pyreloading reloadall reloading types c:\code>C:\Python27\python reloadall.pyreloading reloadall reloading types
With a command-line argument, the tester instead reloads the given module by its name string—here, the benchmark module we coded in Chapter 21. Note that we give a module name in this mode, not a filename (as for import statements, don’t include the .py extension); the script ultimately imports the module using the module search path as usual:
c:\code> reloadall.py pybench
reloading pybench
reloading timeit
reloading itertools
reloading sys
reloading time
reloading gc
reloading os
reloading errno
reloading ntpath
reloading stat
reloading genericpath
reloading copyreg
Perhaps most commonly, we can also deploy this module at the interactive prompt—here, in 3.3 for some standard library modules. Notice how os is imported by tkinter, but tkinter reaches sys before os can (if you want to test this on Python 2.X, substitute Tkinter for tkinter):
>>>from reloadall import reload_all>>>import os, tkinter>>>reload_all(os)# Normal usage mode reloading os reloading ntpath reloading stat reloading sys reloading genericpath reloading errno reloading copyreg >>>reload_all(tkinter)reloading tkinter reloading _tkinter reloading warnings reloading sys reloading linecache reloading tokenize reloading builtins FAILED: <module 'builtins'> reloading re...etc...reloading os reloading ntpath reloading stat reloading genericpath reloading errno...etc...
And finally here is a session that shows the effect of normal versus transitive reloads—changes made to the two nested files are not picked up by reloads, unless the transitive utility is used:
import b # File a.py X = 1 import c # File b.py Y = 2 Z = 3 # File c.py C:\code>py −3>>>import a>>>a.X, a.b.Y, a.b.c.Z(1, 2, 3) # Without stopping Python, change all three files' assignment values and save >>>from imp import reload>>>reload(a)# Built-in reload is top level only <module 'a' from '.\\a.py'> >>>a.X, a.b.Y, a.b.c.Z(111, 2, 3) >>>from reloadall import reload_all>>>reload_all(a)# Normal usage mode reloading a reloading b reloading c >>>a.X, a.b.Y, a.b.c.Z# Reloads all nested modules too (111, 222, 333)
Study the reloader’s code and results for more on its operation. The next section exercises its tools further.
For all the recursion fans in the audience, the following lists an alternative recursive coding for the function in the prior section—it uses a set instead of a dictionary to detect cycles, is marginally more direct because it eliminates a top-level loop, and serves to illustrate recursive function techniques in general (compare with the original to see how this differs). This version also gets some of its work for free from the original, though the order in which it reloads modules might vary if namespace dictionary order does too:
""" reloadall2.py: transitively reload nested modules (alternative coding) """ import types from imp import reload # from required in 3.X from reloadall import status, tryreload, tester def transitive_reload(objects, visited): for obj in objects: if type(obj) == types.ModuleType and obj not in visited: status(obj) tryreload(obj) # Reload this, recur to attrs visited.add(obj) transitive_reload(obj.__dict__.values(), visited) def reload_all(*args): transitive_reload(args, set()) if __name__ == '__main__': tester(reload_all, 'reloadall2') # Test code: reload myself?
As we saw in Chapter 19, there is usually an explicit stack or queue equivalent to most recursive functions, which may be preferable in some contexts. The following is one such transitive reloader; it uses a generator expression to filter out nonmodules and modules already visited in the current module’s namespace. Because it both pops and adds items at the end of its list, it is stack based, though the order of both pushes and dictionary values influences the order in which it reaches and reloads modules—it visits submodules in namespace dictionaries from right to left, unlike the left-to-right order of the recursive versions (trace through the code to see how). We could change this, but dictionary order is arbitrary anyhow.
""" reloadall3.py: transitively reload nested modules (explicit stack) """ import types from imp import reload # from required in 3.X from reloadall import status, tryreload, tester def transitive_reload(modules, visited): while modules: next = modules.pop() # Delete next item at end status(next) # Reload this, push attrs tryreload(next) visited.add(next) modules.extend(x for x in next.__dict__.values() if type(x) == types.ModuleType and x not in visited) def reload_all(*modules): transitive_reload(list(modules), set()) if __name__ == '__main__': tester(reload_all, 'reloadall3') # Test code: reload myself?
If the recursion and nonrecursion used in this example is confusing, see the discussion of recursive functions in Chapter 19 for background on the subject.
To prove that these work the same, let’s test all three of our reloader variants. Thanks to their common testing function, we can run all three from a command line both with no arguments to test the module reloading itself, and with the name of a module to be reloaded listed on the command line (in sys.argv):
c:\code>reloadall.pyreloading reloadall reloading types c:\code>reloadall2.pyreloading reloadall2 reloading types c:\code>reloadall3.pyreloading reloadall3 reloading types
Though it’s hard to see here, we really are testing the individual reloader alternatives—each of these tests shares a common tester function, but passes it the reload_all from its own file. Here are the variants reloading the 3.X tkinter GUI module and all the modules its imports reach:
c:\code>reloadall.py tkinterreloading tkinter reloading _tkinter reloading tkinter._fix...etc...c:\code>reloadall2.py tkinterreloading tkinter reloading tkinter.constants reloading tkinter._fix...etc...c:\code>reloadall3.py tkinterreloading tkinter reloading sys reloading tkinter.constants...etc...
All three work on both Python 3.X and 2.X too—they’re careful to unify prints with formatting, and avoid using version-specific tools (though you must use 2.X module names like Tkinter, and I’m using the 3.3 Windows launcher here to run per Appendix B):
c:\code>py −2 reloadall.pyreloading reloadall reloading types c:\code>py −2 reloadall2.py Tkinterreloading Tkinter reloading _tkinter reloading FixTk...etc...
As usual we can test interactively, too, by importing and calling either a module’s main reload entry point with a module object, or the testing function with a reloader function and module name string:
C:\code>py −3>>>import reloadall, reloadall2, reloadall3>>>import tkinter>>>reloadall.reload_all(tkinter)# Normal use case reloading tkinter reloading tkinter._fix reloading os...etc...>>>reloadall.tester(reloadall2.reload_all, 'tkinter')# Testing utility reloading tkinter reloading tkinter._fix reloading os...etc...>>>reloadall.tester(reloadall3.reload_all, 'reloadall3')# Mimic self-test code reloading reloadall3 reloading types
Finally, if you look at the output of tkinter reloads earlier, you may notice that each of the three variants may produce results in a different order; they all depend on namespace dictionary ordering, and the last also relies on the order in which items are added to its stack. In fact, under Python 3.3, the reload order for a given reloader can vary from run to run. To ensure that all three are reloading the same modules irrespective of the order in which they do so, we can use sets (or sorts) to test for order-neutral equality of their printed messages—obtained here by running shell commands with the os.popen utility we met in Chapter 13 and used in Chapter 21:
>>>import os>>>res1 = os.popen('reloadall.py tkinter').read()>>>res2 = os.popen('reloadall2.py tkinter').read()>>>res3 = os.popen('reloadall3.py tkinter').read()>>>res1[:75]'reloading tkinter\nreloading tkinter.constants\nreloading tkinter._fix\nreload' >>>res1 == res2, res2 == res3(False, False) >>>set(res1) == set(res2), set(res2) == set(res3)(True, True)
Run these scripts, study their code, and experiment on your own for more insight; these are the sort of importable tools you might want to add to your own source code library. Watch for a similar testing technique in the coverage of class tree listers in Chapter 31, where we’ll apply it to passed class objects and extend it further.
Also keep in mind that all three variants reload only modules that were loaded with import statements—since names copied with from statements do not cause a module to be nested and referenced in the importer’s namespace, their containing module is not reloaded. More fundamentally, the transitive reloaders rely on the fact that module reloads update module objects in place, such that all references to those modules in any scope will see the updated version automatically. Because they copy names out, from importers are not updated by reloads—transitive or not—and supporting this may require either source code analysis, or customization of the import operation (see Chapter 22 for pointers).
Tool impacts like this are perhaps another reason to prefer import to from—which brings us to the end of this chapter and part, and the standard set of warnings for this part’s topic.