sqlalchemy.ext.hybrid

Defining Expression Behavior Distinct from Attribute Behavior

Our usage of the & and | bitwise operators above was fortunate, considering our functions operated on two boolean values to return a new one. In many cases, the construction of an in-Python function and a SQLAlchemy SQL expression have enough differences that two separate Python expressions should be defined. The ~sqlalchemy.ext.hybrid decorators define the .hybrid_property.expression modifier for this purpose. As an example we'll define the radius of the interval, which requires the usage of the absolute value function:

from sqlalchemy import func

class Interval(object):
    # ...

    @hybrid_property
    def radius(self):
        return abs(self.length) / 2

    @radius.expression
    def radius(cls):
        return func.abs(cls.length) / 2

Above the Python function abs() is used for instance-level operations, the SQL function ABS() is used via the .func object for class-level expressions:

>>> i1.radius
2

>>> print(Session().query(Interval).filter(Interval.radius > 5))
SELECT interval.id AS interval_id, interval.start AS interval_start,
    interval."end" AS interval_end
FROM interval
WHERE abs(interval."end" - interval.start) / :abs_1 > :param_1

class Interval(object):
    # ...

    @hybrid_property
    def radius(self):
        return abs(self.length) / 2

    # WRONG - the non-matching name will cause this function to be
    # ignored
    @radius.expression
    def radius_expression(cls):
        return func.abs(cls.length) / 2

Defining Setters

Hybrid properties can also define setter methods. If we wanted length above, when set, to modify the endpoint value:

class Interval(object):
    # ...

    @hybrid_property
    def length(self):
        return self.end - self.start

    @length.setter
    def length(self, value):
        self.end = self.start + value

The length(self, value) method is now called upon set:

>>> i1 = Interval(5, 10)
>>> i1.length
5
>>> i1.length = 12
>>> i1.end
17

Allowing Bulk ORM Update

A hybrid can define a custom "UPDATE" handler for when using the _query.Query.update method, allowing the hybrid to be used in the SET clause of the update.

Normally, when using a hybrid with _query.Query.update, the SQL expression is used as the column that's the target of the SET. If our Interval class had a hybrid start_point that linked to Interval.start, this could be substituted directly:

session.query(Interval).update({Interval.start_point: 10})

However, when using a composite hybrid like Interval.length, this hybrid represents more than one column. We can set up a handler that will accommodate a value passed to _query.Query.update which can affect this, using the .hybrid_property.update_expression decorator. A handler that works similarly to our setter would be:

class Interval(object):
    # ...

    @hybrid_property
    def length(self):
        return self.end - self.start

    @length.setter
    def length(self, value):
        self.end = self.start + value

    @length.update_expression
    def length(cls, value):
        return [
            (cls.end, cls.start + value)
        ]

Above, if we use Interval.length in an UPDATE expression as:

session.query(Interval).update(
    {Interval.length: 25}, synchronize_session='fetch')

We'll get an UPDATE statement along the lines of:

UPDATE interval SET end=start + :value

In some cases, the default "evaluate" strategy can't perform the SET expression in Python; while the addition operator we're using above is supported, for more complex SET expressions it will usually be necessary to use either the "fetch" or False synchronization strategy as illustrated above.

    class Interval(object):
        # ...

        def _get(self):
            return self.end - self.start

        def _set(self, value):
            self.end = self.start + value

        def _update_expr(cls, value):
            return [
                (cls.end, cls.start + value)
            ]

        length = hybrid_property(
            fget=_get, fset=_set, update_expr=_update_expr
        )

The Python descriptor protocol does not provide any reliable way for
a descriptor to know what attribute name it was accessed as, and
the UPDATE scheme currently relies upon being able to access the
attribute from an instance by name in order to perform the instance
synchronization step.

Working with Relationships

There's no essential difference when creating hybrids that work with related objects as opposed to column-based data. The need for distinct expressions tends to be greater. The two variants we'll illustrate are the "join-dependent" hybrid, and the "correlated subquery" hybrid.

from sqlalchemy import Column, Integer, ForeignKey, Numeric, String
from sqlalchemy.orm import relationship
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.ext.hybrid import hybrid_property

Base = declarative_base()

class SavingsAccount(Base):
    __tablename__ = 'account'
    id = Column(Integer, primary_key=True)
    user_id = Column(Integer, ForeignKey('user.id'), nullable=False)
    balance = Column(Numeric(15, 5))

class User(Base):
    __tablename__ = 'user'
    id = Column(Integer, primary_key=True)
    name = Column(String(100), nullable=False)

    accounts = relationship("SavingsAccount", backref="owner")

    @hybrid_property
    def balance(self):
        if self.accounts:
            return self.accounts[0].balance
        else:
            return None

    @balance.setter
    def balance(self, value):
        if not self.accounts:
            account = Account(owner=self)
        else:
            account = self.accounts[0]
        account.balance = value

    @balance.expression
    def balance(cls):
        return SavingsAccount.balance

>>> print(Session().query(User, User.balance).
...       join(User.accounts).filter(User.balance > 5000))
SELECT "user".id AS user_id, "user".name AS user_name,
account.balance AS account_balance
FROM "user" JOIN account ON "user".id = account.user_id
WHERE account.balance > :balance_1

>>> from sqlalchemy import or_
>>> print (Session().query(User, User.balance).outerjoin(User.accounts).
...         filter(or_(User.balance < 5000, User.balance == None)))
SELECT "user".id AS user_id, "user".name AS user_name,
account.balance AS account_balance
FROM "user" LEFT OUTER JOIN account ON "user".id = account.user_id
WHERE account.balance <  :balance_1 OR account.balance IS NULL

from sqlalchemy import Column, Integer, ForeignKey, Numeric, String
from sqlalchemy.orm import relationship
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.ext.hybrid import hybrid_property
from sqlalchemy import select, func

Base = declarative_base()

class SavingsAccount(Base):
    __tablename__ = 'account'
    id = Column(Integer, primary_key=True)
    user_id = Column(Integer, ForeignKey('user.id'), nullable=False)
    balance = Column(Numeric(15, 5))

class User(Base):
    __tablename__ = 'user'
    id = Column(Integer, primary_key=True)
    name = Column(String(100), nullable=False)

    accounts = relationship("SavingsAccount", backref="owner")

    @hybrid_property
    def balance(self):
        return sum(acc.balance for acc in self.accounts)

    @balance.expression
    def balance(cls):
        return select(func.sum(SavingsAccount.balance)).\
                where(SavingsAccount.user_id==cls.id).\
                label('total_balance')

>>> print(s.query(User).filter(User.balance > 400))
SELECT "user".id AS user_id, "user".name AS user_name
FROM "user"
WHERE (SELECT sum(account.balance) AS sum_1
FROM account
WHERE account.user_id = "user".id) > :param_1

Building Custom Comparators

The hybrid property also includes a helper that allows construction of custom comparators. A comparator object allows one to customize the behavior of each SQLAlchemy expression operator individually. They are useful when creating custom types that have some highly idiosyncratic behavior on the SQL side.

The example class below allows case-insensitive comparisons on the attribute named word_insensitive:

from sqlalchemy.ext.hybrid import Comparator, hybrid_property
from sqlalchemy import func, Column, Integer, String
from sqlalchemy.orm import Session
from sqlalchemy.ext.declarative import declarative_base

Base = declarative_base()

class CaseInsensitiveComparator(Comparator):
    def __eq__(self, other):
        return func.lower(self.__clause_element__()) == func.lower(other)

class SearchWord(Base):
    __tablename__ = 'searchword'
    id = Column(Integer, primary_key=True)
    word = Column(String(255), nullable=False)

    @hybrid_property
    def word_insensitive(self):
        return self.word.lower()

    @word_insensitive.comparator
    def word_insensitive(cls):
        return CaseInsensitiveComparator(cls.word)

Above, SQL expressions against word_insensitive will apply the LOWER() SQL function to both sides:

>>> print(Session().query(SearchWord).filter_by(word_insensitive="Trucks"))
SELECT searchword.id AS searchword_id, searchword.word AS searchword_word
FROM searchword
WHERE lower(searchword.word) = lower(:lower_1)

The CaseInsensitiveComparator above implements part of the .ColumnOperators interface. A "coercion" operation like lowercasing can be applied to all comparison operations (i.e. eq, lt, gt, etc.) using .Operators.operate:

class CaseInsensitiveComparator(Comparator):
    def operate(self, op, other):
        return op(func.lower(self.__clause_element__()), func.lower(other))

Reusing Hybrid Properties across Subclasses

A hybrid can be referred to from a superclass, to allow modifying methods like .hybrid_property.getter, .hybrid_property.setter to be used to redefine those methods on a subclass. This is similar to how the standard Python @property object works:

class FirstNameOnly(Base):
    # ...

    first_name = Column(String)

    @hybrid_property
    def name(self):
        return self.first_name

    @name.setter
    def name(self, value):
        self.first_name = value

class FirstNameLastName(FirstNameOnly):
    # ...

    last_name = Column(String)

    @FirstNameOnly.name.getter
    def name(self):
        return self.first_name + ' ' + self.last_name

    @name.setter
    def name(self, value):
        self.first_name, self.last_name = value.split(' ', 1)

Above, the FirstNameLastName class refers to the hybrid from FirstNameOnly.name to repurpose its getter and setter for the subclass.

When overriding .hybrid_property.expression and .hybrid_property.comparator alone as the first reference to the superclass, these names conflict with the same-named accessors on the class- level .QueryableAttribute object returned at the class level. To override these methods when referring directly to the parent class descriptor, add the special qualifier .hybrid_property.overrides, which will de- reference the instrumented attribute back to the hybrid object:

class FirstNameLastName(FirstNameOnly):
    # ...

    last_name = Column(String)

    @FirstNameOnly.name.overrides.expression
    def name(cls):
        return func.concat(cls.first_name, ' ', cls.last_name)

Hybrid Value Objects

Note in our previous example, if we were to compare the word_insensitive attribute of a SearchWord instance to a plain Python string, the plain Python string would not be coerced to lower case - the CaseInsensitiveComparator we built, being returned by @word_insensitive.comparator, only applies to the SQL side.

A more comprehensive form of the custom comparator is to construct a Hybrid Value Object. This technique applies the target value or expression to a value object which is then returned by the accessor in all cases. The value object allows control of all operations upon the value as well as how compared values are treated, both on the SQL expression side as well as the Python value side. Replacing the previous CaseInsensitiveComparator class with a new CaseInsensitiveWord class:

class CaseInsensitiveWord(Comparator):
    "Hybrid value representing a lower case representation of a word."

    def __init__(self, word):
        if isinstance(word, basestring):
            self.word = word.lower()
        elif isinstance(word, CaseInsensitiveWord):
            self.word = word.word
        else:
            self.word = func.lower(word)

    def operate(self, op, other):
        if not isinstance(other, CaseInsensitiveWord):
            other = CaseInsensitiveWord(other)
        return op(self.word, other.word)

    def __clause_element__(self):
        return self.word

    def __str__(self):
        return self.word

    key = 'word'
    "Label to apply to Query tuple results"

Above, the CaseInsensitiveWord object represents self.word, which may be a SQL function, or may be a Python native. By overriding operate() and __clause_element__() to work in terms of self.word, all comparison operations will work against the "converted" form of word, whether it be SQL side or Python side. Our SearchWord class can now deliver the CaseInsensitiveWord object unconditionally from a single hybrid call:

class SearchWord(Base):
    __tablename__ = 'searchword'
    id = Column(Integer, primary_key=True)
    word = Column(String(255), nullable=False)

    @hybrid_property
    def word_insensitive(self):
        return CaseInsensitiveWord(self.word)

The word_insensitive attribute now has case-insensitive comparison behavior universally, including SQL expression vs. Python expression (note the Python value is converted to lower case on the Python side here):

>>> print(Session().query(SearchWord).filter_by(word_insensitive="Trucks"))
SELECT searchword.id AS searchword_id, searchword.word AS searchword_word
FROM searchword
WHERE lower(searchword.word) = :lower_1

SQL expression versus SQL expression:

>>> sw1 = aliased(SearchWord)
>>> sw2 = aliased(SearchWord)
>>> print(Session().query(
...                    sw1.word_insensitive,
...                    sw2.word_insensitive).\
...                        filter(
...                            sw1.word_insensitive > sw2.word_insensitive
...                        ))
SELECT lower(searchword_1.word) AS lower_1,
lower(searchword_2.word) AS lower_2
FROM searchword AS searchword_1, searchword AS searchword_2
WHERE lower(searchword_1.word) > lower(searchword_2.word)

Python only expression:

>>> ws1 = SearchWord(word="SomeWord")
>>> ws1.word_insensitive == "sOmEwOrD"
True
>>> ws1.word_insensitive == "XOmEwOrX"
False
>>> print(ws1.word_insensitive)
someword

The Hybrid Value pattern is very useful for any kind of value that may have multiple representations, such as timestamps, time deltas, units of measurement, currencies and encrypted passwords.

Building Transformers

A transformer is an object which can receive a _query.Query object and return a new one. The _query.Query object includes a method .with_transformation that returns a new _query.Query transformed by the given function.

We can combine this with the .Comparator class to produce one type of recipe which can both set up the FROM clause of a query as well as assign filtering criterion.

Consider a mapped class Node, which assembles using adjacency list into a hierarchical tree pattern:

from sqlalchemy import Column, Integer, ForeignKey
from sqlalchemy.orm import relationship
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()

class Node(Base):
    __tablename__ = 'node'
    id = Column(Integer, primary_key=True)
    parent_id = Column(Integer, ForeignKey('node.id'))
    parent = relationship("Node", remote_side=id)

Suppose we wanted to add an accessor grandparent. This would return the parent of Node.parent. When we have an instance of Node, this is simple:

from sqlalchemy.ext.hybrid import hybrid_property

class Node(Base):
    # ...

    @hybrid_property
    def grandparent(self):
        return self.parent.parent

For the expression, things are not so clear. We'd need to construct a _query.Query where we _query.Query.join twice along Node.parent to get to the grandparent. We can instead return a transforming callable that we'll combine with the .Comparator class to receive any _query.Query object, and return a new one that's joined to the Node.parent attribute and filtered based on the given criterion:

from sqlalchemy.ext.hybrid import Comparator

class GrandparentTransformer(Comparator):
    def operate(self, op, other):
        def transform(q):
            cls = self.__clause_element__()
            parent_alias = aliased(cls)
            return q.join(parent_alias, cls.parent).\
                        filter(op(parent_alias.parent, other))
        return transform

Base = declarative_base()

class Node(Base):
    __tablename__ = 'node'
    id = Column(Integer, primary_key=True)
    parent_id = Column(Integer, ForeignKey('node.id'))
    parent = relationship("Node", remote_side=id)

    @hybrid_property
    def grandparent(self):
        return self.parent.parent

    @grandparent.comparator
    def grandparent(cls):
        return GrandparentTransformer(cls)

The GrandparentTransformer overrides the core .Operators.operate method at the base of the .Comparator hierarchy to return a query- transforming callable, which then runs the given comparison operation in a particular context. Such as, in the example above, the operate method is called, given the .Operators.eq callable as well as the right side of the comparison Node(id=5). A function transform is then returned which will transform a _query.Query first to join to Node.parent, then to compare parent_alias using .Operators.eq against the left and right sides, passing into _query.Query.filter:

>>> from sqlalchemy.orm import Session
>>> session = Session()
{sql}>>> session.query(Node).\
...        with_transformation(Node.grandparent==Node(id=5)).\
...        all()
SELECT node.id AS node_id, node.parent_id AS node_parent_id
FROM node JOIN node AS node_1 ON node_1.id = node.parent_id
WHERE :param_1 = node_1.parent_id
{stop}

We can modify the pattern to be more verbose but flexible by separating the "join" step from the "filter" step. The tricky part here is ensuring that successive instances of GrandparentTransformer use the same .AliasedClass object against Node. Below we use a simple memoizing approach that associates a GrandparentTransformer with each class:

class Node(Base):

    # ...

    @grandparent.comparator
    def grandparent(cls):
        # memoize a GrandparentTransformer
        # per class
        if '_gp' not in cls.__dict__:
            cls._gp = GrandparentTransformer(cls)
        return cls._gp

class GrandparentTransformer(Comparator):

    def __init__(self, cls):
        self.parent_alias = aliased(cls)

    @property
    def join(self):
        def go(q):
            return q.join(self.parent_alias, Node.parent)
        return go

    def operate(self, op, other):
        return op(self.parent_alias.parent, other)

{sql}>>> session.query(Node).\
...            with_transformation(Node.grandparent.join).\
...            filter(Node.grandparent==Node(id=5))
SELECT node.id AS node_id, node.parent_id AS node_parent_id
FROM node JOIN node AS node_1 ON node_1.id = node.parent_id
WHERE :param_1 = node_1.parent_id
{stop}

The "transformer" pattern is an experimental pattern that starts to make usage of some functional programming paradigms. While it's only recommended for advanced and/or patient developers, there's probably a whole lot of amazing things it can be used for.