Example: “Private” and “Public” Attributes

The first-cut privacy example shown in Chapter 30 used inheritance to mix in a __setattr__ to catch accesses. Inheritance makes this difficult, however, because differentiating between accesses from inside or outside the class is not straightforward (inside access should be allowed to run normally, and outside access should be restricted). To work around this, the Chapter 30 example requires inheriting classes to use __dict__ assignments to set attributes—an incomplete solution at best.

The version here uses delegation (embedding one object inside another) instead of inheritance; this pattern is better suited to our task, as it makes it much easier to distinguish between accesses inside and outside of the subject class. Attribute accesses from outside the subject class are intercepted by the wrapper layer’s overloading methods and delegated to the class if valid. Accesses inside the class itself (i.e., through self within its methods’ code) are not intercepted and are allowed to run normally without checks, because privacy is not inherited in this version.

The class decorator used here accepts any number of arguments, to name private attributes. What really happens, though, is that the arguments are passed to the Private function, and Private returns the decorator function to be applied to the subject class. That is, the arguments are used before decoration ever occurs; Private returns the decorator, which in turn “remembers” the privates list as an enclosing scope reference.

Speaking of enclosing scopes, there are actually three levels of state retention at work in this code:

This all works fairly naturally, given Python’s scope and namespace rules.

The __setattr__ method in this code relies on an instance object’s __dict__ attribute namespace dictionary in order to set onInstance’s own wrapped attribute. As we learned in the prior chapter, this method cannot assign an attribute directly without looping. However, it uses the setattr built-in instead of __dict__ to set attributes in the wrapped object itself. Moreover, getattr is used to fetch attributes in the wrapped object, since they may be stored in the object itself or inherited by it.

Because of that, this code will work for most classes—including those with “virtual” class-level attributes based on slots, properties, descriptors, and even __getattr__ and its ilk. By assuming a namespace dictionary for itself only and using storage-neutral tools for the wrapped object, the wrapper class avoids limitations inherent in other tools.

For example, you may recall from Chapter 32 that new-style classes with __slots__ may not store attributes in a __dict__ (and in fact may not even have one of these at all). However, because we rely on a __dict__ only at the onInstance level here, and not in the wrapped instance, this concern does not apply. In addition, because setattr and getattr apply to attributes based on both __dict__ and __slots__, our decorator applies to classes using either storage scheme. By the same reasoning, the decorator also applies to new-style properties and similar tools: delegated names will be looked up anew in the wrapped instance, irrespective of attributes of the decorator proxy object itself.

Besides generalizing, this version also makes use of Python’s __X pseudoprivate name mangling feature (which we met in Chapter 31) to localize the wrapped attribute to the proxy control class, by automatically prefixing it with this class’s name. This avoids the prior version’s risk for collisions with a wrapped attribute that may be used by the real, wrapped class, and it’s useful in a general tool like this. It’s not quite “privacy,” though, because the mangled version of the name can be used freely outside the class. Notice that we also have to use the fully expanded name string—'_onInstance__wrapped'— as a test value in __setattr__, because that’s what Python changes it to.

Although this example does implement access controls for attributes of an instance and its classes, it is possible to subvert these controls in various ways—for instance, by going through the expanded version of the wrapped attribute explicitly (bob.pay might not work, but the fully mangled bob._onInstance__wrapped.pay could!). If you have to explicitly try to do so, though, these controls are probably sufficient for normal intended use. Of course, privacy controls can generally be subverted in other languages if you try hard enough (#define private public may work in some C++ implementations, too). Although access controls can reduce accidental changes, much of this is up to programmers in any language; whenever source code may be changed, airtight access control will always be a bit of a pipe dream.

We could again achieve the same results without decorators, by using manager functions or coding the name rebinding of decorators manually; the decorator syntax, however, makes this consistent and a bit more obvious in the code. The chief potential downsides of this and any other wrapper-based approach are that attribute access incurs an extra call, and instances of decorated classes are not really instances of the original decorated class—if you test their type with X.__class__ or isinstance(X, C), for example, you’ll find that they are instances of the wrapper class. Unless you plan to do introspection on objects’ types, though, the type issue is probably irrelevant, and the extra call may apply mostly to development time; as we’ll see later, there are ways to remove decorations automatically if desired.

Like all delegation-based classes that use __getattr__, this decorator works cross-version for normally named or explicitly called attributes only. When run implicitly by built-in operations, operator overloading methods like __str__ and __add__ work differently for new-style classes. Because this code is interpreted as a new-style class in 3.X only, such operations fail to reach an embedded object that defines them when run under this Python line as currently coded.

As we learned in the prior chapter, built-in operations look for operator overloading names in instances for classic classes, but not for new-style classes—for the latter, they skip the instance entirely and begin the search for such methods in classes (technically, in the namespace dictionaries of all classes in the instance’s tree). Hence, the __X__ operator overloading methods implicitly run for built-in operations do not trigger either __getattr__ or __getattribute__ in new-style classes; because such attribute fetches skip our onInstance class’s __getattr__ altogether, they cannot be validated or delegated.

Our decorator’s class is not coded as explicitly new-style (by deriving from object), so it will catch operator overloading methods if run under 2.X as a default classic class. In 3.X, though, because all classes are new-style automatically (and by mandate), such methods will fail if they are implemented by the embedded object—because they are not caught by the proxy, they won’t be passed on.

The most direct workaround in 3.X is to redefine redundantly in onInstance all the operator overloading methods that can possibly be used in wrapped objects. Such extra methods can be added by hand, by tools that partly automate the task (e.g., with class decorators or the metaclasses discussed in the next chapter), or by definition in reusable superclasses. Though tedious—and code-intensive enough to largely omit here—we’ll explore approaches to satisfying this 3.X-only requirement in a moment.

First, though, to see the difference for yourself, try applying the decorator to a class that uses operator overloading methods under 2.X; validations work as before, and both the __str__ method used by printing and the __add__ method run for + invoke the decorator’s __getattr__ and hence wind up being validated and delegated to the subject Person object correctly:

C:\code> c:\python27\python
>>> from access2 import Private
>>> @Private('age')
    class Person:
        def __init__(self):
            self.age = 42
        def __str__(self):
            return 'Person: ' + str(self.age)
        def __add__(self, yrs):
            self.age += yrs

>>> X = Person()
>>> X.age                                   # Name validations fail correctly
TypeError: private attribute fetch: age
>>> print(X)                                # __getattr__ => runs Person.__str__
Person: 42
>>> X + 10                                  # __getattr__ => runs Person.__add__
>>> print(X)                                # __getattr__ => runs Person.__str__
Person: 52

When the same code is run under Python 3.X, though, the implicitly invoked __str__ and __add__ skip the decorator’s __getattr__ and look for definitions in or above the decorator class itself; print winds up finding the default display inherited from the class type (technically, from the implied object superclass in 3.X), and + generates an error because no default is inherited:

C:\code> c:\python33\python
>>> from access2 import Private
>>> @Private('age')
    class Person:
        def __init__(self):
            self.age = 42
        def __str__(self):
            return 'Person: ' + str(self.age)
        def __add__(self, yrs):
            self.age += yrs

>>> X = Person()                            # Name validations still work
>>> X.age                                   # But 3.X fails to delegate built-ins!
TypeError: private attribute fetch: age
>>> print(X)
<access2.accessControl.<locals>.onDecorator.<locals>.onInstance object at ...etc>
>>> X + 10
TypeError: unsupported operand type(s) for +: 'onInstance' and 'int'
>>> print(X)
<access2.accessControl.<locals>.onDecorator.<locals>.onInstance object at ...etc>

Strangely, this occurs only for dispatch from built-in operations; explicit direct calls to overload methods are routed to __getattr__, though clients using operator overloading can’t be expected to do the same:

>>> X.__add__(10)                           # Though calls by name work normally
>>> X._onInstance__wrapped.age              # Break privacy to view result...
52

In other words, this is a matter of built-in operations versus explicit calls; it has little to do with the actual names of the methods involved. Just for built-in operations, Python skips a step for 3.X’s new-style classes.

Using the alternative __getattribute__ method won’t help here—although it is defined to catch every attribute reference (not just undefined names), it is also not run by built-in operations. Python’s property feature, which we met in Chapter 38, won’t help directly here either; recall that properties are automatically run code associated with specific attributes defined when a class is written, and are not designed to handle arbitrary attributes in wrapped objects.

As mentioned earlier, the most straightforward solution under 3.X is to redundantly redefine operator overloading names that may appear in embedded objects in delegation-based classes like our decorator. This isn’t ideal because it creates some code redundancy, especially compared to 2.X solutions. However, it isn’t an impossibly major coding effort; can be automated to some extent with tools or superclasses; suffices to make our decorator work in 3.X; and may allow operator overloading names to be declared Private or Public too, assuming overloading methods trigger the failIf test internally.

def accessControl(failIf):
    def onDecorator(aClass):
        class onInstance:
            def __init__(self, *args, **kargs):
                self.__wrapped = aClass(*args, **kargs)

            # Intercept and delegate built-in operations specifically
            def __str__(self):
                return str(self.__wrapped)
            def __add__(self, other):
                return self.__wrapped + other          # Or getattr(x, '__add__')(y)
            def __getitem__(self, index):
                return self.__wrapped[index]           # If needed
            def __call__(self, *args, **kargs):
                return self.__wrapped(*args, **kargs)  # If needed
            # plus any others needed

            # Intercept and delegate by-name attribute access generically
            def __getattr__(self, attr): ...
            def __setattr__(self, attr, value): ...
        return onInstance
    return onDecorator

class BuiltinsMixin:
    def __add__(self, other):
        return self.__class__.__getattr__(self, '__add__')(other)
    def __str__(self):
        return self.__class__.__getattr__(self, '__str__')()
    def __getitem__(self, index):
        return self.__class__.__getattr__(self, '__getitem__')(index)
    def __call__(self, *args, **kargs):
        return self.__class__.__getattr__(self, '__call__')(*args, **kargs)
    # plus any others needed

def accessControl(failIf):
    def onDecorator(aClass):
        class onInstance(BuiltinsMixin):
            ...rest unchanged...
            def __getattr__(self, attr): ...
            def __setattr__(self, attr, value): ...


class BuiltinsMixin:
    def __add__(self, other):
        return self._wrapped + other                    # Assume a _wrapped
    def __str__(self):                                  # Bypass __getattr__
        return str(self._wrapped)
    def __getitem__(self, index):
        return self._wrapped[index]
    def __call__(self, *args, **kargs):
        return self._wrapped(*args, **kargs)
    # plus any others needed

def accessControl(failIf):
    def onDecorator(aClass):
        class onInstance(BuiltinsMixin):
            ...and use self._wrapped instead of self.__wrapped...
            def __getattr__(self, attr): ...
            def __setattr__(self, attr, value): ...

class BuiltinsMixin:
    def reroute(self, attr, *args, **kargs):
        return self.__class__.__getattr__(self, attr)(*args, **kargs)

    def __add__(self, other):
        return self.reroute('__add__', other)
    def __str__(self):
        return self.reroute('__str__')
    def __getitem__(self, index):
        return self.reroute('__getitem__', index)
    def __call__(self, *args, **kargs):
        return self.reroute('__call__', *args, **kargs)
    # plus any others needed

class BuiltinsMixin:
    class ProxyDesc(object):                                     # object for 2.X
        def __init__(self, attrname):
            self.attrname = attrname
        def __get__(self, instance, owner):
            return getattr(instance._wrapped, self.attrname)     # Assume a _wrapped

    builtins = ['add', 'str', 'getitem', 'call']                 # Plus any others
    for attr in builtins:
        exec('__%s__ = ProxyDesc("__%s__")' % (attr, attr))

    __add__ = ProxyDesc("__add__")
    __str__ = ProxyDesc("__str__")
    ...etc...

Adding support for operator overloading methods is required of interface proxies in general, to delegate calls correctly. In our specific privacy application, though, it also raises some additional design choices. In particular, privacy of operator overloading methods differs per implementation:

We’ll leave final policy here a TBD, but some interface proxies might prefer to allow __X__ operator names to always pass unchecked when delegated.

In the general case, though, a substantial amount of extra code is required to accommodate 3.X’s new-style classes as delegation proxies—in principle, every operator overloading method that is no longer dispatched as a normal instance attribute automatically will need to be defined redundantly in a general tool class like this privacy decorator. This is why this extension is omitted in our code: there are potentially more than 50 such methods! Because all its classes are new-style, delegation-based code is more difficult—though not necessarily impossible—in Python 3.X.

Although redundantly defining operator overloading methods in wrappers is probably the most straightforward workaround to Python 3.X dilemma outlined in the prior section, it’s not necessarily the only one. We don’t have space to explore this issue much further here, so deeper investigation will have to be relegated to suggested exercise. Because one dead-end alternative illustrates class concepts well, though, it merits a brief mention.

One downside of the privacy example is that instance objects are not truly instances of the original class—they are instances of the wrapper instead. In some programs that rely on type testing, this might matter. To support such cases, we might try to achieve similar effects by inserting a __getattribute__ and a __setattr__ method into the original class, to catch every attribute reference and assignment made on its instances. These inserted methods would pass valid requests up to their superclass to avoid loops, using the techniques we studied in the prior chapter. Here is the potential change to our class decorator’s code:

# Method insertion: rest of access2.py code as before

def accessControl(failIf):
    def onDecorator(aClass):
        def getattributes(self, attr):
            trace('get:', attr)
            if failIf(attr):
                raise TypeError('private attribute fetch: ' + attr)
            else:
                return object.__getattribute__(self, attr)

        def setattributes(self, attr, value):
            trace('set:', attr)
            if failIf(attr):
                raise TypeError('private attribute change: ' + attr)
            else:
                return object.__setattr__(self, attr, value)

        aClass.__getattribute__ = getattributes
        aClass.__setattr__ = setattributes               # Insert accessors
        return aClass                                    # Return original class
    return onDecorator

This alternative addresses the type-testing issue but suffers from others. For one thing, this decorator can be used by new-style class clients only: because __getattribute__ is a new-style-only tool (as is this __setattr__ coding), decorated classes in 2.X must use new-style derivation, which may or may not be appropriate for their goals. In fact, the set of classes supported is even further limited: inserting methods will break clients that are already using a __setattr__ or __getattribute__ of their own.

Worse, this scheme does not address the built-in operation attributes issue described in the prior section, because __getattribute__ is also not run in these contexts. In our case, if Person had a __str__ it would be run by print operations, but only because it was actually present in that class. As before, the __str__ attribute would not be routed to the inserted __getattribute__ method generically—printing would bypass this method altogether and call the class’s __str__ directly.

Although this is probably better than not supporting operator overloading methods in a wrapped object at all (barring redefinition, at least), this scheme still cannot intercept and validate __X__ methods, making it impossible for any of them to be private. Whether operator overloading methods should be private is another matter, but this structure precludes the possibility.

Much worse, because this nonwrapper approach works by adding a __getattribute__ and __setattr__ to the decorated class, it also intercepts attribute accesses made by the class itself and validates them the same as accesses made from outside. In other words, the class’s own method won’t be able to use its private names either! This is a showstopper for the insertion approach.

In fact, inserting these methods this way is functionally equivalent to inheriting them, and implies the same constraints as our original Chapter 30 privacy code. To know whether an attribute access originated inside or outside the class, our methods might need to inspect frame objects on the Python call stack. This might ultimately yield a solution—implementing private attributes as properties or descriptors that check the stack and validate for outside accesses only, for example—but it would slow access further, and is far too dark a magic for us to explore here. (Descriptors seem to make all things possible, even when they shouldn’t!)

While interesting, and possibly relevant for some other use cases, this method insertion technique doesn’t meet our goals. We won’t explore this option’s coding pattern further here because we will study class augmentation techniques in the next chapter, in conjunction with metaclasses. As we’ll see there, metaclasses are not strictly required for changing classes this way, because class decorators can often serve the same role.