sqlalchemy.ext.mutable

Establishing Mutability on Scalar Column Values

A typical example of a "mutable" structure is a Python dictionary. Following the example introduced in :ref:`types_toplevel`, we begin with a custom type that marshals Python dictionaries into JSON strings before being persisted:

from sqlalchemy.types import TypeDecorator, VARCHAR
import json

class JSONEncodedDict(TypeDecorator):
    "Represents an immutable structure as a json-encoded string."

    impl = VARCHAR

    def process_bind_param(self, value, dialect):
        if value is not None:
            value = json.dumps(value)
        return value

    def process_result_value(self, value, dialect):
        if value is not None:
            value = json.loads(value)
        return value

The usage of json is only for the purposes of example. The sqlalchemy.ext.mutable extension can be used with any type whose target Python type may be mutable, including .PickleType, _postgresql.ARRAY, etc.

When using the sqlalchemy.ext.mutable extension, the value itself tracks all parents which reference it. Below, we illustrate a simple version of the .MutableDict dictionary object, which applies the .Mutable mixin to a plain Python dictionary:

from sqlalchemy.ext.mutable import Mutable

class MutableDict(Mutable, dict):
    @classmethod
    def coerce(cls, key, value):
        "Convert plain dictionaries to MutableDict."

        if not isinstance(value, MutableDict):
            if isinstance(value, dict):
                return MutableDict(value)

            # this call will raise ValueError
            return Mutable.coerce(key, value)
        else:
            return value

    def __setitem__(self, key, value):
        "Detect dictionary set events and emit change events."

        dict.__setitem__(self, key, value)
        self.changed()

    def __delitem__(self, key):
        "Detect dictionary del events and emit change events."

        dict.__delitem__(self, key)
        self.changed()

The above dictionary class takes the approach of subclassing the Python built-in dict to produce a dict subclass which routes all mutation events through __setitem__. There are variants on this approach, such as subclassing UserDict.UserDict or collections.MutableMapping; the part that's important to this example is that the .Mutable.changed method is called whenever an in-place change to the datastructure takes place.

We also redefine the .Mutable.coerce method which will be used to convert any values that are not instances of MutableDict, such as the plain dictionaries returned by the json module, into the appropriate type. Defining this method is optional; we could just as well created our JSONEncodedDict such that it always returns an instance of MutableDict, and additionally ensured that all calling code uses MutableDict explicitly. When .Mutable.coerce is not overridden, any values applied to a parent object which are not instances of the mutable type will raise a ValueError.

Our new MutableDict type offers a class method ~.Mutable.as_mutable which we can use within column metadata to associate with types. This method grabs the given type object or class and associates a listener that will detect all future mappings of this type, applying event listening instrumentation to the mapped attribute. Such as, with classical table metadata:

from sqlalchemy import Table, Column, Integer

my_data = Table('my_data', metadata,
    Column('id', Integer, primary_key=True),
    Column('data', MutableDict.as_mutable(JSONEncodedDict))
)

Above, ~.Mutable.as_mutable returns an instance of JSONEncodedDict (if the type object was not an instance already), which will intercept any attributes which are mapped against this type. Below we establish a simple mapping against the my_data table:

from sqlalchemy import mapper

class MyDataClass(object):
    pass

# associates mutation listeners with MyDataClass.data
mapper(MyDataClass, my_data)

The MyDataClass.data member will now be notified of in place changes to its value.

There's no difference in usage when using declarative:

from sqlalchemy.ext.declarative import declarative_base

Base = declarative_base()

class MyDataClass(Base):
    __tablename__ = 'my_data'
    id = Column(Integer, primary_key=True)
    data = Column(MutableDict.as_mutable(JSONEncodedDict))

Any in-place changes to the MyDataClass.data member will flag the attribute as "dirty" on the parent object:

>>> from sqlalchemy.orm import Session

>>> sess = Session()
>>> m1 = MyDataClass(data={'value1':'foo'})
>>> sess.add(m1)
>>> sess.commit()

>>> m1.data['value1'] = 'bar'
>>> assert m1 in sess.dirty
True

The MutableDict can be associated with all future instances of JSONEncodedDict in one step, using ~.Mutable.associate_with. This is similar to ~.Mutable.as_mutable except it will intercept all occurrences of MutableDict in all mappings unconditionally, without the need to declare it individually:

MutableDict.associate_with(JSONEncodedDict)

class MyDataClass(Base):
    __tablename__ = 'my_data'
    id = Column(Integer, primary_key=True)
    data = Column(JSONEncodedDict)

class MyMutableType(Mutable):
    def __getstate__(self):
        d = self.__dict__.copy()
        d.pop('_parents', None)
        return d

class MutableDict(Mutable, dict):
    # ....

    def __getstate__(self):
        return dict(self)

    def __setstate__(self, state):
        self.update(state)

from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import event

Base = declarative_base()

class MyDataClass(Base):
    __tablename__ = 'my_data'
    id = Column(Integer, primary_key=True)
    data = Column(MutableDict.as_mutable(JSONEncodedDict))

@event.listens_for(MyDataClass.data, "modified")
def modified_json(instance):
    print("json value modified:", instance.data)

Establishing Mutability on Composites

Composites are a special ORM feature which allow a single scalar attribute to be assigned an object value which represents information "composed" from one or more columns from the underlying mapped table. The usual example is that of a geometric "point", and is introduced in :ref:`mapper_composite`.

As is the case with .Mutable, the user-defined composite class subclasses .MutableComposite as a mixin, and detects and delivers change events to its parents via the .MutableComposite.changed method. In the case of a composite class, the detection is usually via the usage of Python descriptors (i.e. @property), or alternatively via the special Python method __setattr__(). Below we expand upon the Point class introduced in :ref:`mapper_composite` to subclass .MutableComposite and to also route attribute set events via __setattr__ to the .MutableComposite.changed method:

from sqlalchemy.ext.mutable import MutableComposite

class Point(MutableComposite):
    def __init__(self, x, y):
        self.x = x
        self.y = y

    def __setattr__(self, key, value):
        "Intercept set events"

        # set the attribute
        object.__setattr__(self, key, value)

        # alert all parents to the change
        self.changed()

    def __composite_values__(self):
        return self.x, self.y

    def __eq__(self, other):
        return isinstance(other, Point) and \
            other.x == self.x and \
            other.y == self.y

    def __ne__(self, other):
        return not self.__eq__(other)

The .MutableComposite class uses a Python metaclass to automatically establish listeners for any usage of _orm.composite that specifies our Point type. Below, when Point is mapped to the Vertex class, listeners are established which will route change events from Point objects to each of the Vertex.start and Vertex.end attributes:

from sqlalchemy.orm import composite, mapper
from sqlalchemy import Table, Column

vertices = Table('vertices', metadata,
    Column('id', Integer, primary_key=True),
    Column('x1', Integer),
    Column('y1', Integer),
    Column('x2', Integer),
    Column('y2', Integer),
    )

class Vertex(object):
    pass

mapper(Vertex, vertices, properties={
    'start': composite(Point, vertices.c.x1, vertices.c.y1),
    'end': composite(Point, vertices.c.x2, vertices.c.y2)
})

Any in-place changes to the Vertex.start or Vertex.end members will flag the attribute as "dirty" on the parent object:

>>> from sqlalchemy.orm import Session

>>> sess = Session()
>>> v1 = Vertex(start=Point(3, 4), end=Point(12, 15))
>>> sess.add(v1)
>>> sess.commit()

>>> v1.end.x = 8
>>> assert v1 in sess.dirty
True

class Point(MutableComposite):
    # other Point methods
    # ...

    def coerce(cls, key, value):
        if isinstance(value, tuple):
            value = Point(*value)
        elif not isinstance(value, Point):
            raise ValueError("tuple or Point expected")
        return value

class Point(MutableComposite):
    # ...

    def __getstate__(self):
        return self.x, self.y

    def __setstate__(self, state):
        self.x, self.y = state