Auditing with Reified Transactions in Datomic

Datomic is the database that underlies the Shortcut application, and one of the great things about Datomic is its implementation of the reified transaction. In addition to the application domain entities stored in the database, transactions themselves are entities that can be queried and inspected in order to establish what happened and when.

Shortcut uses reified transactions to provide organizations with a historical record of the actions their members took, and in turn, help our users organize, query, and manage their work. Shortcut allows you to ask and get answers for questions like:

• “Who worked on this?”
• “When was that completed?”
• “Who assigned me all of this work?”

Prerequisites

This post assumes some familiarity with Clojure and Datomic. There are online resources for learning more about Datomic and more specifically Datomic Transactions.

All examples below are preceded by this require function call for using the Datomic API:

user> (require '[datomic.api :as d])

Now, let’s get technical…

Inspecting the Transaction Entity

Given an entity-id for a transaction (you can get this a number of different ways including from within the return value of the d/transact command), this is what you will see upon inspection:

user> (d/touch (d/entity db 13199994644064)){:db/id 13199994644064, :db/txInstant #inst "2017-04-12T19:03:32.011-00:00"}

The transaction entity has two keys:

1. :db/id  — the transaction id that we started with
2. :db/txInstant  — the timestamp of the transaction

What does Shortcut Audit?

When using Shortcut, users take actions within the context of an organization. We add the following two attributes to every transaction to indicate both the actor and the context:

• :audit/user  — the actor who carried out the transaction
• :audit/org  — the context in which the transaction occurred

We can later query these transactions in order to provide a detailed history of actions taken by a specific user working within a specific organization.

How to add Audit Attributes

To add fields to the transaction entity, create a tempid in the :db.part/txpartition and add the audit fields using that tempid:

(defn transact-wrapper [connection user-id org-id tx-data]  (let [tx-tempid (d/tempid :db.part/tx)        tx-data-audit [[:db/add tx-tempid :audit/user user-id]                       [:db/add tx-tempid :audit/org org-id]]]    @(d/transact connection (concat tx-data tx-data-audit))))

Inspecting a transaction transacted by this function yields two more fields than the example above:

{:db/id 13199994641376 :db/txInstant #inst "2017-04-12T14:03:13.400-00:00" :audit/user {:db/id 19999186272840} :audit/org {:db/id 19999186045777}}

And, with this audit information attached to the transaction entities, we can now query transactions in order to generate a stream of activity feed history for a user within an organization:

(defn find-tx-ids [db user-id org-id]  (d/q '[:find [?tx-id ...]         :in $ ?user-id ?org-id         :where         [?tx-id :audit/user ?user-id]         [?tx-id :audit/org ?org-id]]       db user-id org-id))

Interpreting an individual transaction is out of scope for this post, but it is by no means trivial. You could use additional audit fields to provide more detail or inspect the individual datoms contained in the transaction using d/log.

Conclusion

As a project management tool, Shortcut has several features that rely on historical transaction data. Through the use of reified transactions, Shortcut uses audit information to track many things: the actor, the context, the action taken, and more.

We’ve found this valuable as we continue to push out features that give Shortcut users a bigger picture of their progress over time.

A special thanks to Edward Wible and Lucas Cavalcanti at Nubank for their 2014 ClojureConj talk “Exploring Four Datomic Superpowers” which inspired us to use reified transactions.