Happy belated new year! The Datomic team was among the six hundred participants at Clojure/Conj 2025. Here are some highlights.

Datomic in Computational biology

Benjamin Kamphaus’s Power Tools for Translational Data Science (transcript) was a more philosophical talk. It weaves historical linguistics, computational phylogenetics, constraint satisfaction, and the biology of rare cancers into a parable on why it matters how we build our tools.

It builds on his prior talks about his data science toolkit, Clojure Where it Counts: Tidying Data Science Workflows (with Pier Federico) and Building a Unified Cancer Immunotherapy Data Library (with Lacey Kitch), both of which dive more into the technical advantages that Clojure and Datomic provide.

Thanks to the organizers, speakers, and everyone who made the Conj such a vibrant time. We hope to see you there next year!

The Amazing Day of Datomic!

What is Jepsen?

Jepsen Analysis Outcome

Conclusion

The Datomic team is excited to announce the launch of Datomic Local under the Apache License 2.0. Datomic Local is a library that is both embeddable and redistributable. It is a lightweight version of Datomic, designed to be easily embedded within applications and freely redistributed. Datomic Local supports the Client API, making it an excellent choice for small single-process applications and for testing systems that are based on the Datomic Client API.

FAQ

Nubank is excited to announce today that Datomic Cloud is also now available free of licensing fees, and the Datomic binaries are being released under the Apache 2.0 license. Starting with this version, Datomic Cloud is now easier to get. To set up Datomic Cloud, check the instructions on how to get started. We aim to make Datomic even more useful to more people in more situations. We are looking forward to expanding Datomic and Clojure worldwide!

FAQ

Nubank is excited to announce today that we are making all editions of Datomic available free of licensing fees. The Datomic binaries are being released under the Apache 2.0 license and will be readily available for direct download and use via Maven Central - no signup required. Datomic Cloud will be available on AWS Marketplace with no additional software cost.

FAQ

If you have an application that is struggling with performance, it might be due to poor data locality, the physical distribution of relevant information. A query that is searching for sparsely distributed data will find few results per storage read. Conversely, queries can efficiently slurp relevant data when it is densely colocated. Datomic’s partitions allow applications to group entities together and gain control over locality. Entities that share a partition will be colocated in the E-leading indexes, EAVT and AEVT. Because each partition requires a name and installation prior to its use, it can become cumbersome to build schemes that use many partitions.

(d/transact conn [{:db/id (d/tempid (d/implicit-part 55))
                   :movie/title "The Great Gatsby"}])

(d/transact conn [{:db/id "movie"
                   :movie/title "The Great Gatsby"
                   :db/force-partition {"movie" (d/implicit-part 55)}}])

(d/transact conn
            [{:db/id    "order"
              :order/id 123
              :order/items ["item1" "item2"]}

             {:db/id     "item1"
              :item/name "cereal"
              :item/sku  "A150"}

             {:db/id     "item2"
              :item/name "milk"
              :item/sku  "T770"}

             {:db/force-partition {"order" (d/implicit-part 15555)}
              :db/match-partition {"item1" "order"
                                   "item2" "order"}}])

Today’s release of Datomic On-Prem, and coming soon in Cloud, includes a new feature: query-stats.

{:find  [(count ?artist-name)]
 :where [[?release :release/name ?release-name]
         [?release :release/artists ?artist]
         [?artist :artist/name ?artist-name]]}

{:ret         [[4587]],
 :query-stats {:query  {:find  [(count ?artist-name)],
                        :where [[?release :release/name ?release-name]
                                [?release :release/artists ?artist]
                                [?artist :artist/name ?artist-name]]},
               :phases [{:sched   (([?release :release/name ?release-name]
                                    [?release :release/artists ?artist]
                                    [?artist :artist/name ?artist-name])),
                         :clauses [{:clause [?release :release/name ?release-name], :rows-in 0, :rows-out 11434, :binds-in (), :binds-out [?release], :expansion 11434, :warnings {:unbound-vars #{?release ?release-name}}}
                                   {:clause [?release :release/artists ?artist], :rows-in 11434, :rows-out 4600, :binds-in [?release], :binds-out [?artist]}
                                   {:clause [?artist :artist/name ?artist-name], :rows-in 4600, :rows-out 4587, :binds-in [?artist], :binds-out [?artist-name]}]}]}}

Datomic Cloud 884-9095 is a major new release offering more instance options, lower pricing, API Gateway automation for ions and clients, and more.

The Datomic Knowledgebase has been launched to provide a place to ask questions, provide answers, discuss and make feature requests.

We are thrilled to announce that Cognitect is joining the Nubank family of companies. This is the next step in a long relationship, and opens new opportunities for Clojure and Datomic worldwide. Please read the full story over on the Cognitect Blog.

The analytics metaschema specifies a mapping from Datomic entities and attributes to dynamic SQL tables and columns, as shown in the example below:

• Watch the video. 
• Read the Cloud docs.
• Read the On-Prem docs.

• If you declare a composite tuple, Datomic will automatically populate the tuple from existing attributes. 
• You can also define your own homogeneous or heterogeneous tuple types that you populate as you see fit.

• Attribute predicates are declared in schema and constrain the values taken on by a single attribute, in all contexts.  For example, you could limit a test/grade attribute to the range 0-100.
• Entity specs comprise lists of required attributes and/or predicates of the entity and the post-transaction db. You can use entity specs to ensure properties across different attributes of an entity or even across entities.  For example, you might enforce that a game player's score/lowest must be less than or equal to their score/highest.  You must explicitly call for entity specs in transactions in which you want them to apply.

[:find ?artist-name ?release-name
 :keys artist release
 :where [?release :release/name ?release-name]
 [?release :release/artists ?artist]
 [?artist :artist/name ?artist-name]]

#{{:artist "George Jones" :release "With Love"}
 {:artist "Shocking Blue" :release "Hello Darkness / Pickin' Tomatoes"} 
 {:artist "Junipher Greene" :release "Friendship"}
 ...}

• Sign up for Datomic Cloud on AWS Marketplace.
• Read the new return maps docs.

1. Get the latest release of Datomic Cloud.
2. Work through the HTTP Direct Tutorial.

• Get the latest release of Datomic.
• Read the Valcache docs. 

Rationale

Get Started

Datomic Cloud and AWS CloudWatch

• Collect and track CloudWatch Metrics -- variables that measure the behavior of your system.
• Configure CloudWatch Alarms to notify operations or take other automated steps when potential problems arise.
• Monitor, store, and search CloudWatch Logs across all your AWS resources.
• Create CloudWatch Dashboards that provide a single overview for monitoring your systems.

Production Dashboard

• Focus on your application
• Leverage your Datomic Cloud cluster compute resources and data locality
• Extend Datomic transactions and queries with your own logic
• Connect to the broader AWS cloud via Lambda events
• Service web consumers with API Gateway
• Scale Datomic and your app together
• Deliver on AWS with high agility

Datomic Cloud on the AWS Marketplace

• authentication and authorization
• network-level access
• how the bastion works
• some example deployments

Authentication and Authorization

Network-Level Access

• run an EC2 instance in the Datomic VPC and in the Datomic applications security group
• peer another VPC with the Datomic VPC 
• configure a VPN Connection

How the Bastion Works

1. run the Datomic socks proxy script on your local machine
2. add a proxy port argument when creating a system client
3. the Datomic client sees the proxy port argument and connects to the socks proxy 
4. the socks proxy forwards encrypted SSH traffic to the bastion 
5. the bastion forwards Datomic client protocol traffic to Datomic

A Concrete Example

Conclusion

• my videos of setting up a Datomic Cloud system
• the access control docs
• the bastion docs

• How does it work?
• What do I have to do?
• Hey wait a minute! What about serializability?
• Is AWS cool?
• So what else have you got?

How Does It Work?

What Do I Have To Do?

Hey Wait a Minute! What About Serializability?

Datomic is always transactional, fully serialized, and consistent in both the ACID and CAP senses. Don't waste your life writing code to compensate for partial failures and subtle concurrency bugs when you could be making your application better and shipping it faster.

So how does that square with shared-nothing cluster nodes? The answer is simple: The nodes use DynamoDB to serialize all writes per database.

1. This is not a master/follower system like Datomic On-Prem and many other databases – nobody is tracking mastership and there are no failover intervals.
2. This should not be confused with parallel multi-writer systems such as Cassandra. Write availability is governed by the availability of DynamoDB conditional writes and strongly-consistent reads.

Is AWS Cool?

• CloudFormation automates creation and updates the cluster 
• Application Load Balancers distribute requests transparently to clients
• Auto Scaling Groups and health checks maintain a healthy cluster
• DynamoDB conditional writes ensure transaction semantics

So What Else Have You Got?

For More Information:

• High Availability
• the Production and Solo Topologies

• Easy: Datomic Cloud is an Amazon Marketplace AMI. 
  • click through a CloudFormation wizard and you can be up and running in minutes
  • start with the Solo Topology (including AWS instance) for about $1/day, billed hourly
  • itemized on your regular AWS bill
• Integrated: Datomic Cloud is closely integrated with AWS, automatically providing:
  • encryption at rest with key management by KMS
  • clustered high availability behind Application Load Balancers (ALBs)
  • elastic read autoscaling via autoscaling groups without sharding
  • redundant storage across multiple highly-available, fault-tolerant storages: DynamoDB, S3, and EFS
  • CloudWatch metrics and dashboards
• Powerful: Datomic Cloud delivers the full Datomic information model
  
  • ACID transactions
  • Datalog query with joins, recursion, and rules
  • a universal relation supporting row, column, entity, and graph access
  • indelible, chronological storage so you can track data over time

Datomic Cloud is accessible through the latest release of the Datomic Client API. To learn more, you can:

• see how to set up in minutes
• try it yourself
• learn more about the architecture
• review the differences between Cloud and On-Prem

Datomic Cloud on the AWS Marketplace

;; pull expression
'[[:track/_artists :limit 1]]

=> #:track{:_artists
 [#:db{:id 17592188757937}]}

;; pull pattern
'[{[:track/_artists :limit 1] [:track/name]}]

=> #:track{:_artists [#:track{:name "Black Dog"}]}

• you are targeting an environment that does not support symbolic names, so you need a string instead of a keyword key
• you do not want to expose the direction of navigation (e.g. the underscore in :track/_artists)
• your consumers are expecting a different name

;; pull expression
'[{[:track/_artists :limit 1 :as "Tracks"]
 [[:track/name :as "Name"]]}]

=> {"Tracks" [{"Name" "Black Dog"}]}

1.  Prod is not like dev.

2.  Grow your schema, and never break it.

• growth is providing more schema
• breakage is removing schema, or changing the meaning of existing schema.

3. The database is the source of truth.

4.  Growing is adding.

• adding new attributes to an existing 'type'
• adding new types
• adding relationships between types

5.  Never remove a name.

6.  Never reuse a name.

7.  Use aliases.

• In iteration 1, users of your system are identified by their email with an attribute named :user/id. 
• In iteration 2, you discover that users sometimes have non-email identifiers for users and that you want to store a user's email even when not using the email as an identifier. In short, you wish that :user/id was named :user/primary-email.

8.  Namespace all names.

9.  Annotate your schema.

• you could annotate names that are not recommended for new programs with a :schema/deprecated flag, or you could get fancier still with :schema/deprecated-at or :schema/deprecated-because. Note that such deprecated names are still never removed (Rule Five).
• you could provide :schema/see-also or :schema/see-instead pointers to more current conventions. 

• add N new columns
• (optional) add a :schema/split-into attribute on the original column whose value is the new columns, and possibly even the recipe for the split

10. Plan for accretion.

Are these rules specific to a particular database?

In Context

• In addition to the peer model, Datomic now includes a Client API suitable for smaller, short lived processes, e.g. microservices.
• The various tiers of the Datomic Pro license model have been simplified to a single license with no restriction on peer count.
• We have introduced an Enterprise license tier for users who need customized pricing, support, or licensing terms.
• Tempids and explicit partitions are now optional, simplifying code for the many programs that do not care about them.
• Schema install and update are now implicit, and do not require explicit :db.install/attribute or :db.update/attribute datoms.

Tempid and Partition Defaults

(d/q '[:find ?tx
 :in $ ?name ?street
 :where
 [?e :person/name ?name]
 [?e :person/street ?street ?tx true]]
 (d/history (d/db conn)) "Joe" "Broadway")

(d/pull (d/db conn) '[:db/txInstant] 13194139534317)

(d/q '[:find ?e ?a ?v ?tx ?op
 :in ?log ?tx
 :where [(tx-data ?log ?tx)[[?e ?a ?v _ ?op]]]]
 (d/log conn) 13194139534317)

;; result:
#{[17592186045420 64 "2nd" 13194139534317 true]
 [17592186045419 64 "Broadway" 13194139534317 true]
 [13194139534317 50 #inst"1983-01-01T00:00:00.000-00:00" 13194139534317 true]
 [17592186045420 64 "Elm" 13194139534317 false]
 [17592186045419 64 "1st" 13194139534317 false]}

(d/pull (d/db conn) '[*] 17592186045420)

;; result:
{:db/id 17592186045420, :person/name "Mary", :person/street "2nd"}

• What products are in inventory?
• What users have accounts on the system?
• What tests have been run?

• rapid development against a memory database
• schema definition
• hierarchical selection with pull
• Datalog query
• strong consistency
• retractions and history

• Designing Schema
• Transaction Strategies
• Building and Optimizing Queries
• Time in Datomic

• Negation, via the new not and not-join clauses
• Disjunction (or) without using rules, via the new or and or-join clauses
• Required rule bindings
• Improved optimization of range predicates

Negation

[:find ?name
 :where [?eid :artist/name ?name]
 (not [?eid :artist/country :country/CA])]

[:find (count ?artist) .
 :where [?artist :artist/name]
 (not-join [?artist]
 [?release :release/artists ?artist]
 [?release :release/year 1970])]

Disjunction

[:find (count ?medium) .
 :where (or [?medium :medium/format :medium.format/vinyl7]
 [?medium :medium/format :medium.format/vinyl10]
 [?medium :medium/format :medium.format/vinyl12]
 [?medium :medium/format :medium.format/vinyl])]

[:find (count ?artist) .
 :where (or [?artist :artist/type :artist.type/group]
 (and [?artist :artist/type :artist.type/person]
 [?artist :artist/gender :artist.gender/female]))]

All Together

[:find ?name
 :where [?eid :artist/name ?name]
 (not (or [?eid :artist/country :country/US])
 [?eid :artist/country :country/CA])]

[(short-track ?a ?t ?len ?max)
 [?t :track/artists ?a]
 [?t :track/duration ?len]
 [(< ?len ?max)]]

• invoke the rule in a context where no variables are bound yet, finding all possible relationships
• invoke the rule with all variables bound, as a predicate check
• any combination of variable bindings between these two extremes 

(short-track [?a ?max] ?t ?len)

Optimization of Range Predicates

[:find (count ?artist) .
 :in $ ?max-year
 :where [?artist :artist/startYear ?year]
 [(< ?year ?max-year)]]

Summary

Pull API vs. Entity API

Wildcards

;; Clojure API
(d/pull db '[*] led-zeppelin)

;; Java API
db.pull("[*]", ledZeppelin)

;; result
{:artist/sortName "Led Zeppelin", 
 :artist/name "Led Zeppelin", 
 :artist/type {:db/id 17592186045746}, 
 :artist/country {:db/id 17592186045576}, 
 :artist/gid #uuid "678d88b2-87b0-403b-b63d-5da7465aecc3", 
 :artist/endDay 25, 
 :artist/startYear 1968, 
 :artist/endMonth 9, 
 :artist/endYear 1980, 
 :db/id 17592186050305}

Attributes

;; pattern
[:artist/name :artist/gid]

;; input 
led-zeppelin

;; result
{:artist/gid #uuid "678d88b2-87b0-403b-b63d-5da7465aecc3", 
 :artist/name "Led Zeppelin"}

;; pattern
[:artist/_country]

;; input
greatBritain

;; result
{:artist/_country [{:db/id 17592186045751} 
 {:db/id 17592186045755} 
 ...]}

Components

;; pattern
[:release/media]

;; input
darkSideOfTheMoon

;; result
 {:release/media
 [{:db/id 17592186121277,
 :medium/format {:db/id 17592186045741},
 :medium/position 1,
 :medium/trackCount 10,
 :medium/tracks
 [{:db/id 17592186121278,
 :track/duration 68346,
 :track/name "Speak to Me",
 :track/position 1,
 :track/artists [{:db/id 17592186046909}]}
 {:db/id 17592186121279,
 :track/duration 168720,
 :track/name "Breathe",
 :track/position 2,
 :track/artists [{:db/id 17592186046909}]}
 {:db/id 17592186121280,
 :track/duration 230600,
 :track/name "On the Run",
 :track/position 3,
 :track/artists [{:db/id 17592186046909}]}
 ...]}]}

Map Specifications

;; pattern
[:track/name {:track/artists [:db/id :artist/name]}]

;; input
ghostRiders

;; result
{:track/artists [{:db/id 17592186048186, :artist/name "Bob Dylan"}
 {:db/id 17592186049854, :artist/name "George Harrison"}],
 :track/name "Ghost Riders in the Sky"}

 ;; pattern
[{:release/media
 [{:medium/tracks
 [:track/name {:track/artists [:artist/name]}]}]}]

;; input
concertForBanglaDesh

;; result
 [{:medium/tracks
 [{:track/artists
 [{:artist/name "Ravi Shankar"} {:artist/name "George Harrison"}],
 :track/name "George Harrison / Ravi Shankar Introduction"}
 {:track/artists [{:artist/name "Ravi Shankar"}],
 :track/name "Bangla Dhun"}]}
 {:medium/tracks
 [{:track/artists [{:artist/name "George Harrison"}],
 :track/name "Wah-Wah"}
 {:track/artists [{:artist/name "George Harrison"}],
 :track/name "My Sweet Lord"}
 {:track/artists [{:artist/name "George Harrison"}],
 :track/name "Awaiting on You All"}
 {:track/artists [{:artist/name "Billy Preston"}],
 :track/name "That's the Way God Planned It"}]
 ...]}

Try It Out

A Motivating Example

What "Delete" Means

What "Last" Means

What "Entry" Means

Retraction in Code

addTribbles = [[':db/id': tempid(':db.part/user'),
 ':item/id': '0042-TRBL',
 ':item/description': 'Tribble: a low maintenance pet.',
 ':item/count': 999]];
conn.transact(addTribbles).get();

Finding the Problem

db = conn.db();
db.entity([':item/id', '0042-TRBL']);

q('''[:find ?aname ?v ?tx ?added
 :in $ ?e
 :where [?e ?a ?v ?tx ?added]
 [?a :db/ident ?aname]]''',
 db,
 [':item/id', '0042-TRBL']);

Searching The Log

log = conn.log();
q('''[:find ?e ?aname ?v ?tx ?added
 :in $ ?log ?tx
 :where [(tx-data ?log ?tx) [[?e ?a ?v _ ?added]]]
 [?a :db/ident ?aname]]''',
 db,
 log,
 db.basisT());

Fixing the Problem

errDoc = 'Error correction entry. We do not sell Tribbles.';
weDontSellTribbles = 
[[':db/add', [':item/id', '0042-TRBL'], ':item/count', 0],
 [':db/add', tempid(':db.part/tx'), ':db/doc', errDoc]];
conn.transact(weDontSellTribbles).get();

retractTribbles = 
[[':db.fn/retractEntity', [':item/id', '0042-TRBL']],
 [':db/add', 
 tempid(':db.part/tx'), 
 ':corrected/by', [':manager/email', 'jdoe@example.com']]];
conn.transact(retractTribbles).get();

hist = conn.db().history();
q('''[:find ?aname ?v ?tx ?added
 :in $ ?e
 :where [?e ?a ?v ?tx ?added]
 [?a :db/ident ?aname]]''',
 hist,
 [':item/id', '0042-TRBL']);

Modeling Time: Concerns and Tradeoffs

Performance Concerns

Not Everything Needs History

Excision

Clean History

Wrapping Up

Some of the benefits:

• Reduced work to be done per indexing job

• rename attributes
• rename your own programmatic identities (uses of :db/ident)
• add or remove indexes 
• add or remove uniqueness constraints
• change attribute cardinality
• change whether history is retained for an attribute
• change whether an attribute is treated as a component

Motivation

Implementation

• Support for all storages
• A perpetual license with 12 months of updates included
• Support for the full Datomic programming model
• Datomic Console included with download

Exploring Schema

Query

Entities

Transactions

Indexes

And More

The Transaction Report Queue

final BlockingQueue queue = conn.txReportQueue();

while (true) {
 final Map tx = queue.take();
 Collection results = q(byAttribute, tx.get(TX_DATA), attrid);
 for (Iterator it = results.iterator(); it.hasNext(); ) {
 printList(it.next());
 }
}

• It deals with a domain with which nearly everyone is familiar
• It is of decent size: 60,438 labels; 664,226 artists; 1,035,592 album releases; and 13,233,625 recorded tracks
• It comprises a good number of entities, attributes, and relationships
• It is fun to play with, query, and explore

Schema

• We omit any notion of Work
• We combine Track, Tracklist and Recording into simply "track"
• We renamed Release group to "abstractRelease"

Abstract Release vs. Release vs. Medium

Relationship Diagram

Entities

Getting Started

Getting the Data

 # 2.8 GB, md5 4e7d254c77600e68e9dc71b1a2785c53
 wget http://s3.amazonaws.com/mbrainz/datomic-mbrainz-backup-20130611.tar

 # this takes a while
 tar -xvf datomic-mbrainz-backup-20130611.tar

 # takes a while, but prints progress -- ~150,000 segments in restore
 bin/datomic restore-db file:datomic-mbrainz-backup-20130611 datomic:free://localhost:4334/mbrainz

Getting the Code

 git clone git@github.com:Datomic/mbrainz-sample.git
 cd mbrainz-sample

Running the examples

From Java

 -Xmx2g -server

From Clojure

 # from the root of the mbrainz-sample repo
 lein repl

Thanks

Why Components?

{:db/id #db/id[:db.part/db]
 :db/ident :order/lineItems
 :db/isComponent true
 :db/valueType :db.type/ref
 :db/cardinality :db.cardinality/many
 :db.install/_attribute :db.part/db}

Creating Components

productQuery = '''[:find ?e
 :in $ ?v
 :where [?e :product/description ?v]]''';

(chocolate, whisky) = ['Expensive Chocolate', 'Cheap Whisky'].collect {
 q(productQuery, conn.db(), it)[0][0];
}
===> [17592186045454, 17592186045455]

order = [['order/lineItems': [['lineItem/product': chocolate,
 'lineItem/quantity': 1,
 'lineItem/price': 48.00],
 ['lineItem/product': whisky,
 'lineItem/quantity': 2,
 'lineItem/price': 38.00]],
 'db/id': tempid()]];

conn.transact(order).get();

Touching Components

ordersByProductQuery = '''
[:find ?e
 :in $ ?productDesc
 :where [?e :order/lineItems ?item]
 [?item :lineItem/product ?prod]
 [?prod :product/description ?productDesc]]''';

• from the provided productDesc input to to the product entity ?prod
• from ?prod to the order item ?item
• from ?item to the order ?e

qe = { query, db, Object[] more ->
 db.entity(q(query, db, *more)[0][0])
}

order = qe(ordersByProductQuery, db, 'Expensive Chocolate');

===> {:db/id 17592186045457}

order.touch();
===> {:order/lineItems #{{:lineItem/product #, 
 :lineItem/price 38.00M, 
 :lineItem/quantity 2, 
 :db/id 17592186045459} 
 {:lineItem/product #, 
 :lineItem/price 48.00M, 
 :lineItem/quantity 1, 
 :db/id 17592186045458}}, 
 :db/id 17592186045457}

Retracting Components

conn.transact([[":db.fn/retractEntity", order[":db/id"]]]).get();

q('''[:find (count ?e)
 :where [?e :order/lineItems]]''',
 db);
===> []

q('''[:find (count ?e)
 :where [?e :product/description]]''',
 db);
===> [[2]]

Conclusion

Why Groovy

aList = ['John', 'Doe'];
aMap = [firstName: 'John', 
 lastName: 'Doe'];

lastNames = people.collect { it['lastName'] }

• Of the popular expressive languages that target the JVM, Groovy's syntax is most similar to Java's.

Transactions

conn.transact([[op, entityId, attributeName, value]]);

:hereIsAnUnqualifiedName
:the.namespace.comes.before/theName

conn.transact([[':db/add', newUserId, ':user/firstName', 'John']]);

// create an entity with two attributes (map form)
conn.transact([[':db/id': newUserId,
 ':user/firstName': 'John',
 ':user/lastName': 'Doe']]);

// create an entity with two attributes (list form)
conn.transact([[':db/add' newUserId, ':user/firstName', 'John'],
 [':db/add' newUserId, ':user/lastName', 'Doe']]);

newUser = [[':db/id': newUserId,
 ':user/email': 'john@example.com',
 ':user/firstName': 'John',
 ':user/lastName': 'Doe']];

upvoteStories = storyIds.collect {
 [':db/add', newUserId, ':user/upVotes', it]
}

conn.transact(upvoteAllStories + newUser);

Query

q('''[:find ?e 
 :where [?e :user/email]]''', db);

q('''[:find ?e
 :in $ ?email
 :where [?e :user/email ?email]]''',
 db, 'editor@example.com');

q('''[:find ?e
 :in $ [?email ...]
 :where [?e :user/email ?email]]''',
 db, ['editor@example.com', 'stuarthalloway@datomic.com']);

q('''[:find ?comment
 :in $ ?email
 :where [?user :user/email ?email]
 [?comment :comment/author ?user]]''',
 db, 'editor@example.com')

Background

Motivation

Tools

Conclusion

Reading the Code

A Simple Schema

Get Connected

Listing 1: Getting Connected

(require
 '[datomic.api :as d]
 '[datomic.samples.news :as news])

 (def uri "datomic:mem://news")
 (def conn (news/setup-sample-db-1 uri))

Get a Database Value

Listing 2: Getting a Database Value

(def db (d/db conn))

A first query

Listing 3: Finding All Users

(d/q '[:find ?e
 :where [?e :user/email]]
 db)
  

=> #{[17592186045424] [17592186045425]}

• As query is the most commonly used function in Datomic, it has the terse name d/q.
• The first argument is a query expression. The :where clause indicates what you want to find, "those entities ?e that have a :user/email attribute." The :find clause tells which variables to return.
• The second argument is the database value we obtained in the previous step.

A simple join

Listing 4: Finding a Specific User

(d/q '[:find ?e
 :in $ ?email
 :where [?e :user/email ?email]]
 db
 "editor@example.com")
  

=> #{[17592186045425]}

A database join

Listing 5: Finding a User's Comments

(d/q '[:find ?comment
 :in $ ?email
 :where [?user :user/email ?email]
 [?comment :comment/author ?user]]
 db
 "editor@example.com")

=> #{[17592186045451]
 [17592186045450]
 ... }

Aggregates

Listing 6: Returning an Aggregate

(d/q '[:find (count ?comment)
 :in $ ?email
 :where [?user :user/email ?email]
 [?comment :comment/author ?user]]
 db
 "editor@example.com")
  

=> [[10]]

More joins

Listing 7: Multiple Joins

(d/q '[:find (count ?comment)
 :where [?comment :comment/author]
 [?commentable :comments ?comment]
 [?commentable :user/email]]
 db)

=> []

Schema-aware joins

Listing 8: A Schema Query

(d/q '[:find ?attr-name
 :where [?ref :comments]
 [?ref ?attr]
 [?attr :db/ident ?attr-name]]
 db)

=> #{[:story/title]
 [:comment/body]
 [:story/url]
 [:comment/author]
 [:comments]}

Entities

Listing 9: Finding an Entity ID

(def editor-id (->> (d/q '[:find ?e
 :in $ ?email
 :where [?e :user/email ?email]]
 db
 "editor@example.com")
 ffirst))

Listing 10: Getting an Entity

(def editor (d/entity db editor-id))

Listing 11: A Lazy Entity

editor

=> {:db/id 17592186045425}

Listing 12: Requesting an Attribute

(:user/firstName editor)

=> "Edward"

Listing 13: Touching an Entity

(d/touch editor)
 => {:user/firstName "Edward",
 :user/lastName "Itor",
 :user/email "editor@example.com",
 :db/id 17592186045425}

Are Entities ORM?

Listing 14: Navigating Backwards

(-> editor :comment/_author)

=> [{:db/id 17592186045441}
 {:db/id 17592186045443}
 ... ]

Listing 15: Navigating Deeper

(->> editor :comment/_author (mapcat :comments))

=> ({:db/id 17592186045448}
 {:db/id 17592186045450}
 ...)

Time travel

• the transaction entity tx that created the datom
• the relative time, t of the transaction
• the clock time :db/txInstant of the transaction

Listing 16: Querying for a Transaction

(def txid (->> (d/q '[:find ?tx
 :in $ ?e
 :where [?e :user/firstName _ ?tx]]
 db
 editor-id)
 ffirst))

Listing 17: Converting Transaction to T

(d/tx->t txid)
=> 1023

Listing 18: Getting a Tx Instant

(-> (d/entity (d/db conn) txid) :db/txInstant)

=> #inst "2013-02-20T16:27:11.788-00:00"

Listing 19: Going Back in Time

(def older-db (d/as-of db (dec txid)))
  

(:user/firstName (d/entity older-db editor-id))

=> "Ed"

Auditing

Listing 20: Querying Across All Time

(def hist (d/history db))

(->> (d/q '[:find ?tx ?v ?op
 :in $ ?e ?attr
 :where [?e ?attr ?v ?tx ?op]]
 hist
 editor-id
 :user/firstName)
 (sort-by first))
  

=> ([13194139534319 "Ed" true]
 [13194139534335 "Ed" false]
 [13194139534335 "Edward" true])

Everything is Data

Listing 21: Querying Plain Java Data

(d/q '[:find ?e
 :where [?e :user/email]]
 [[1 :user/email "jdoe@example.com"]
 [1 :user/firstName "John"]
 [2 :user/email "jane@example.com"]])

=> #{[1] [2]}

• In-process memory
• Transactor-local dev/free mode
• SQL
• DynamoDB
• Riak
• Couchbase
• Infinispan

codeq

Backstory

Short Story

• Track change at the program unit level (e.g. function and method definitions)
• Query your programs and libraries declaratively, with the same cognitive units and names you use while programming
• Query across repos

Long Story

How it works

In Detail

Put on a happy interface

(def rules
 '[[(node-files ?n ?f) [?n :node/object ?f] [?f :git/type :blob]]
 [(node-files ?n ?f) [?n :node/object ?t] [?t :git/type :tree] 
 [?t :tree/nodes ?n2] (node-files ?n2 ?f)]
 [(object-nodes ?o ?n) [?n :node/object ?o]]
 [(object-nodes ?o ?n) [?n2 :node/object ?o] [?t :tree/nodes ?n2] (object-nodes ?t ?n)]
 [(commit-files ?c ?f) [?c :commit/tree ?root] (node-files ?root ?f)]
 [(commit-codeqs ?c ?cq) (commit-files ?c ?f) [?cq :codeq/file ?f]]
 [(file-commits ?f ?c) (object-nodes ?f ?n) [?c :commit/tree ?n]]
 [(codeq-commits ?cq ?c) [?cq :codeq/file ?f] (file-commits ?f ?c)]])

• Find all the files referenced by a commit
• Find all the codeqs referenced by a commit
• Find all the commits including a file
• Find all the commits including a codeq

(d/q '[:find ?src (min ?date)
 :in $ % ?name 
 :where
 [?n :code/name ?name]
 [?cq :clj/def ?n]
 [?cq :codeq/code ?cs]
 [?cs :code/text ?src]
 [?cq :codeq/file ?f]
 (file-commits ?f ?c)
 (?c :commit/authoredAt ?date)]
 db rules "datomic.codeq.core/commit")

Q&A

• A peer service can serve more than one database, against more than one storage (Pro)
• Clients get the same kind of data-driven API as do peers
• Clients can get repeatable queries, and multiple queries with the same basis
• Clients get the time-travel as-of and since capabilities, raw access to indexes and ranges
• Clients can issue multi-datasource queries, and pass data to query
• Clients can get a push feed of transaction events, via a Server-Sent Events source

• Login to your ElastiCache management console and choose "Launch Cache Cluster"
• Complete the wizard. You can accept all defaults.
• Back at the console, select "Cache Security Groups":
  
• ... and authorize your Datomic system's security group (default is "datomic"):
  
  
• Back at the console, select "Cache Clusters" and copy the node endpoints string:
  
  
• Paste the node endpoints into your transactor properties file:

• You can get a shared cache of arbitrary size - many deployments will be able to fit their entire database in memcached if desired. 
• If you are using a storage that is not otherwise distributed (e.g. unclustered PostgreSQL), the memcached tier can both almost entirely remove the read load on the single server and distribute it.
• Even when using a distributed storage like DynamoDB, the memcached tier can reduce your read provisioning and increase speed.
• Developers can set up a small memcached daemon locally so their DB will always feel 'hot' across process restarts.
• The memcached tier will enable hybrid strategies where the peers, transactors and memcached are all local but the storage is remote (e.g. DynamoDB).

• We've added Datomic Free Edition - it's free and redistributable
• Datomic Pro is licensed software.
• It is offered with a perpetual license.
• Maintenance (updates and support) for the first 12 months is included.
• Maintenance in subsequent years is ~50% of the license fee.
• Pricing is yearly, and up front on web site.
• Pricing is based upon the number of processes (transactors + peers) using the software in production in your organization.
• Development and testing usage doesn't count against your license limits

• Datomic Free Edition is ... free!
• The system supports transactor-local storage
• The peer library includes a memory database and Datomic Datalog
• The Free transactor and peers are freely redistributable 
• The transactor supports 2 simultaneous peers