Step 7 (Final): Storing Objects in a Database

Object persistence is implemented by three standard library modules, available in every Python:

We met these modules very briefly in Chapter 9 when we studied file basics. They provide powerful data storage options. Although we can’t do them complete justice in this tutorial or book, they are simple enough that a brief introduction is enough to get you started.

Pickling and shelves are somewhat advanced topics, and we won’t go into all their details here; you can read more about them in the standard library manuals, as well as application-focused books such as the Programming Python follow-up text. This is all simpler in Python than in English, though, so let’s jump into some code.

Let’s write a new script that throws objects of our classes onto a shelve. In your text editor, open a new file we’ll call makedb.py. Since this is a new file, we’ll need to import our classes in order to create a few instances to store. We used from to load a class at the interactive prompt earlier, but really, as with functions and other variables, there are two ways to load a class from a file (class names are variables like any other, and not at all magic in this context):

import person                                    # Load class with import
bob = person.Person(...)                         # Go through module name

from person import Person                        # Load class with from
bob = Person(...)                                # Use name directly

We’ll use from to load in our script, just because it’s a bit less to type. To keep this simple, copy or retype in our new script the self-test lines from person.py that make instances of our classes, so we have something to store (this is a simple demo, so we won’t worry about the test-code redundancy here). Once we have some instances, it’s almost trivial to store them on a shelve. We simply import the shelve module, open a new shelve with an external filename, assign the objects to keys in the shelve, and close the shelve when we’re done because we’ve made changes:

# File makedb.py: store Person objects on a shelve database

from person import Person, Manager               # Load our classes
bob = Person('Bob Smith')                        # Re-create objects to be stored
sue = Person('Sue Jones', job='dev', pay=100000)
tom = Manager('Tom Jones', 50000)

import shelve
db = shelve.open('persondb')                     # Filename where objects are stored
for obj in (bob, sue, tom):                      # Use object's name attr as key
    db[obj.name] = obj                           # Store object on shelve by key
db.close()                                       # Close after making changes

Notice how we assign objects to the shelve using their own names as keys. This is just for convenience; in a shelve, the key can be any string, including one we might create to be unique using tools such as process IDs and timestamps (available in the os and time standard library modules). The only rule is that the keys must be strings and should be unique, since we can store just one object per key, though that object can be a list, dictionary, or other object containing many objects itself.

In fact, the values we store under keys can be Python objects of almost any sort—built-in types like strings, lists, and dictionaries, as well as user-defined class instances, and nested combinations of all of these and more. For example, the name and job attributes of our objects could be nested dictionaries and lists as in earlier incarnations in this book (though this would require a bit of redesign to the current code).

That’s all there is to it—if this script has no output when run, it means it probably worked; we’re not printing anything, just creating and storing objects in a file-based database.

C:\code> makedb.py

At this point, there are one or more real files in the current directory whose names all start with “persondb”. The actual files created can vary per platform, and just as in the built-in open function, the filename in shelve.open() is relative to the current working directory unless it includes a directory path. Wherever they are stored, these files implement a keyed-access file that contains the pickled representation of our three Python objects. Don’t delete these files—they are your database, and are what you’ll need to copy or transfer when you back up or move your storage.

You can look at the shelve’s files if you want to, either from Windows Explorer or the Python shell, but they are binary hash files, and most of their content makes little sense outside the context of the shelve module. With Python 3.X and no extra software installed, our database is stored in three files (in 2.X, it’s just one file, persondb, because the bsddb extension module is preinstalled with Python for shelves; in 3.X, bsddb is an optional third-party open source add-on).

For example, Python’s standard library glob module allows us to get directory listings in Python code to verify the files here, and we can open the files in text or binary mode to explore strings and bytes:

>>> import glob
>>> glob.glob('person*')
['person-composite.py', 'person-department.py', 'person.py', 'person.pyc',
'persondb.bak', 'persondb.dat', 'persondb.dir']

>>> print(open('persondb.dir').read())
'Sue Jones', (512, 92)
'Tom Jones', (1024, 91)
'Bob Smith', (0, 80)

>>> print(open('persondb.dat','rb').read())
b'\x80\x03cperson\nPerson\nq\x00)\x81q\x01}q\x02(X\x03\x00\x00\x00jobq\x03NX\x03\x00
...more omitted...

This content isn’t impossible to decipher, but it can vary on different platforms and doesn’t exactly qualify as a user-friendly database interface! To verify our work better, we can write another script, or poke around our shelve at the interactive prompt. Because shelves are Python objects containing Python objects, we can process them with normal Python syntax and development modes. Here, the interactive prompt effectively becomes a database client:

>>> import shelve
>>> db = shelve.open('persondb')                 # Reopen the shelve

>>> len(db)                                      # Three 'records' stored
3
>>> list(db.keys())                              # keys is the index
['Sue Jones', 'Tom Jones', 'Bob Smith']          # list() to make a list in 3.X

>>> bob = db['Bob Smith']                        # Fetch bob by key
>>> bob                                          # Runs __repr__ from AttrDisplay
[Person: job=None, name=Bob Smith, pay=0]

>>> bob.lastName()                               # Runs lastName from Person
'Smith'

>>> for key in db:                               # Iterate, fetch, print
        print(key, '=>', db[key])

Sue Jones => [Person: job=dev, name=Sue Jones, pay=100000]
Tom Jones => [Manager: job=mgr, name=Tom Jones, pay=50000]
Bob Smith => [Person: job=None, name=Bob Smith, pay=0]

>>> for key in sorted(db):
        print(key, '=>', db[key])                # Iterate by sorted keys

Bob Smith => [Person: job=None, name=Bob Smith, pay=0]
Sue Jones => [Person: job=dev, name=Sue Jones, pay=100000]
Tom Jones => [Manager: job=mgr, name=Tom Jones, pay=50000]

Notice that we don’t have to import our Person or Manager classes here in order to load or use our stored objects. For example, we can call bob’s lastName method freely, and get his custom print display format automatically, even though we don’t have his Person class in our scope here. This works because when Python pickles a class instance, it records its self instance attributes, along with the name of the class it was created from and the module where the class lives. When bob is later fetched from the shelve and unpickled, Python will automatically reimport the class and link bob to it.

The upshot of this scheme is that class instances automatically acquire all their class behavior when they are loaded in the future. We have to import our classes only to make new instances, not to process existing ones. Although a deliberate feature, this scheme has somewhat mixed consequences:

Shelves also have well-known limitations (the database suggestions at the end of this chapter mention a few of these). For simple object storage, though, shelves and pickles are remarkably easy-to-use tools.

Now for one last script: let’s write a program that updates an instance (record) each time it runs, to prove the point that our objects really are persistent—that their current values are available every time a Python program runs. The following file, updatedb.py, prints the database and gives a raise to one of our stored objects each time. If you trace through what’s going on here, you’ll notice that we’re getting a lot of utility “for free”—printing our objects automatically employs the general __repr__ overloading method, and we give raises by calling the giveRaise method we wrote earlier. This all “just works” for objects based on OOP’s inheritance model, even when they live in a file:

# File updatedb.py: update Person object on database

import shelve
db = shelve.open('persondb')               # Reopen shelve with same filename

for key in sorted(db):                     # Iterate to display database objects
    print(key, '\t=>', db[key])            # Prints with custom format

sue = db['Sue Jones']                      # Index by key to fetch
sue.giveRaise(.10)                         # Update in memory using class's method
db['Sue Jones'] = sue                      # Assign to key to update in shelve
db.close()                                 # Close after making changes

Because this script prints the database when it starts up, we have to run it at least twice to see our objects change. Here it is in action, displaying all records and increasing sue’s pay each time it is run (it’s a pretty good script for sue...something to schedule to run regularly as a cron job perhaps?):

C:\code> updatedb.py
Bob Smith       => [Person: job=None, name=Bob Smith, pay=0]
Sue Jones       => [Person: job=dev, name=Sue Jones, pay=100000]
Tom Jones       => [Manager: job=mgr, name=Tom Jones, pay=50000]

C:\code> updatedb.py
Bob Smith       => [Person: job=None, name=Bob Smith, pay=0]
Sue Jones       => [Person: job=dev, name=Sue Jones, pay=110000]
Tom Jones       => [Manager: job=mgr, name=Tom Jones, pay=50000]

C:\code> updatedb.py
Bob Smith       => [Person: job=None, name=Bob Smith, pay=0]
Sue Jones       => [Person: job=dev, name=Sue Jones, pay=121000]
Tom Jones       => [Manager: job=mgr, name=Tom Jones, pay=50000]

C:\code> updatedb.py
Bob Smith       => [Person: job=None, name=Bob Smith, pay=0]
Sue Jones       => [Person: job=dev, name=Sue Jones, pay=133100]
Tom Jones       => [Manager: job=mgr, name=Tom Jones, pay=50000]

Again, what we see here is a product of the shelve and pickle tools we get from Python, and of the behavior we coded in our classes ourselves. And once again, we can verify our script’s work at the interactive prompt—the shelve’s equivalent of a database client:

C:\code> python
>>> import shelve
>>> db = shelve.open('persondb')             # Reopen database
>>> rec = db['Sue Jones']                    # Fetch object by key
>>> rec
[Person: job=dev, name=Sue Jones, pay=146410]
>>> rec.lastName()
'Jones'
>>> rec.pay
146410

For another example of object persistence in this book, see the sidebar in Chapter 31 titled Why You Will Care: Classes and Persistence. It stores a somewhat larger composite object in a flat file with pickle instead of shelve, but the effect is similar. For more details and examples for both pickles and shelves, see also Chapter 9 (file basics) and Chapter 37 (3.X string tool changes), other books, and Python’s manuals.