Calypso: A MongoDB Data Schema in Scala

Introduction

To apply Domain-Driven Design, DDD Aggregate, and Transactional outbox on MongoDB, the author's team built an open-source library: calypso, for working with BSON.

This post is for readers who aim for modern development practices and share the author's enthusiasm for Scala 3.

Ready for some discoveries? Welcome to the world of functional programming and dependable work with schema-on-read.

Plan

domain modeling with ADTs;
converting the domain model to BSON;
working with the calypso library on Scala 3;
MongoDB data schema evolution;
schema-on-read practices.

How do you find a useful model?

"All models are wrong, some are useful." George Box

Business process automation starts with a model. The storage schema, design patterns, and component decomposition matter too, but if the model is bad, a clear solution is unlikely to follow.

Using a toy problem, let's do some modeling. The task: deactivate a user while recording exactly when it happened.

import java.time.Instant

case class Person(id: Int, deactivated: Boolean, deactivatedAt: Option[Instant])

This model solves the task: it can express both an active user and a deactivated one, with a time. The representation has a flaw — there is a way to create a user in an invalid state.

Person(1, deactivated = true, None)

The deactivation time is missing. You can catch this in code review and write unit tests, but there is a more elegant way.

"Making Invalid State Unrepresentable." Yaron Minsky

Express the user's status with a Scala 3 enum.

enum Status:
  case Active
  case Deactivated(t: Instant)

case class Person(id: Int, status: Status)

This approach rules out some errors at the type level and makes testing those scenarios impossible. Unlike the first model, you cannot deactivate a user without a time.

Person(1, Status.Active)
Person(1, Status.Deactivated(Instant.EPOCH))

The example above is algebraic data types used for modeling. Expressing composition of attributes with "and" is possible in many programming languages — that is a struct, class, or tuple. In general it is called a product. Conversely, expressing the "or" idea, which is used for the Active and Deactivated variants, is rare. In Java and Kotlin it is a sealed class, in Scala 2 a sealed trait. Scala 3 offers enum, clearly expressing the idea. The terms coproduct, sum type, and tagged union are also used.

The key point to take from this: algebraic data types give a coherent, unambiguous model and eliminate a class of errors. The compiler plays the role of unit tests, not letting errors be expressed in code.

Functional programming patterns let you express part of the logic at the type level. How do you keep those advantages when moving to a storage model?

How do you store such a model in a database?

MongoDB was chosen for several reasons:

the absence of a relational model removes the need for an ORM;
Change Streams suit implementing Transactional outbox;
it scales horizontally.

The main use of MongoDB in the author's team is storing the state of domain entities, such as Person from the examples above. Simplified: the storage needs read and write operations by key, transformations over state happen in the application, and state never changes partially — only as a whole.

BSON

MongoDB's data model looks like JSON, but works with binary data. The serialized value of Person in BSON:

{"id": 1, "status": {"tag": "Deactivated", value: {"t": 0}}}

BSON represents data types like Int, String, Array, Object. It is a rich toolkit, but not as expressive as the types of the domain model. Transformations are needed to get BSON from Scala types such as List, case class, or enum. The MongoDB Scala Driver, ReactiveMongo, MongoLess, shapeless-reactivemongo, Pure BSON, medeia, and circe-bson libraries solve this problem. But none of them combine the qualities that matter to the author's team at the same time, such as:

a convenient interface for encoding case classes and enums;
no compile-time or runtime reflection;
binary data support.

The previous section found the advantages of algebraic data types in modeling; from this follows intensive use of enum in the code. You need a convenient way to encode such structures in BSON.

Reflection removes boilerplate, but refactoring such as renaming fields breaks reading existing BSON data in Scala.

The JSON subset that circe-bson offers does not work with binary data, even though circe has a great interface for encoding case classes.

To get everything at once, the team built the calypso library for type-safe conversions between BSON and Scala.

Calypso

The library provides Encoder and Decoder abstractions with constructors. Also combinators for transforming them. Encoder and Decoder in calypso are a way to translate between the domain model and MongoDB's storage model.

import org.bson.BsonValue

trait Encoder[A]:
  def encode(a: A): BsonValue

Conceptually this is a function from A to BsonValue. Let's look at the new Person example, how to encode a case class:

import org.bson.BsonValue
import ru.m2.calypso.Encoder
import ru.m2.calypso.syntax.*

case class Person(id: Int, name: String)
object Person:
  given Encoder[Person] = Encoder.forProduct2("id", "name")(p => (p.id, p.name))

val bson: BsonValue = Person(1, "Алиса").asBson // {"id": 1, "name": "Алиса"}

In the Encoder description, the field names id and name are given explicitly — this is done so that renaming fields in Scala does not affect the names in BSON.

The forProduct2 constructor means creating an Encoder for a product, or a case class with two attributes. It is also called "arity two".

Given instances are a Scala 3 construct. In Scala 2 it is called implicit val. The idea is to provide a canonical value of Encoder[Person] that the compiler will use automatically, for example when calling .asBson on a Person.

An Encoder is a theorem, its implementation is a proof. If an Encoder for Person exists, a value can be converted to BSON. The pattern of a parameterized interface with a canonical implementation, as in the example above, is called a type class. It lets you attach behavior to data structures without changing their implementation. It is used heavily in Haskell and Scala.

A concise interface like forProduct2 is made possible by the conditional implicits mechanism in Scala. If an Encoder for Int and String exists, the compiler derives an Encoder for their pair. Calypso supports the base types and some collections from the Scala standard library.

An example of encoding an enum:

import org.bson.BsonValue
import ru.m2.calypso.Encoder
import ru.m2.calypso.syntax.*

enum AorB:
  case A(i: Int)
  case B(s: String)

object AorB:
  given Encoder[A] = Encoder.forProduct1("i")(_.i)
  given Encoder[B] = Encoder.forProduct1("s")(_.s)
  given Encoder[AorB] = Encoder.forCoproduct:
    case a: A => "A" -> a.asBson
    case b: B => "B" -> b.asBson

val aBson: BsonValue = (AorB.A(42): AorB).asBson
// {"tag": "A", "value": {"i": 42}}

Not as elegant as a case class, but the solution works. With an Encoder for each variant of AorB — that is A and B — we can convert them to BSON. You need to specify the constructor name when using forCoproduct. This is needed to distinguish the variants at decoding time. The team tried different interfaces for encoding enums and settled on this one: it turned out the clearest to read. If you have a different approach, share it in the comments.

Decoder works in a similar way. The type-directed programming or compile-time derived codecs mechanism in the example above gives a reliable solution without using macros or runtime reflection.

How do you evolve the model with this approach, given that MongoDB has "no schema"?

Data schema evolution

New use cases change the model; the user now has a surname:

case class Person(id: Int, name: String, surname: String)

New codecs are needed to support the new attributes and stay compatible with existing data.

import ru.m2.calypso.Decoder

object Person:
  val decodePersonV1: Decoder[Person] =
    Decoder.forProduct2("id", "name")((id, name) => Person(id, name, ""))
  val decodePersonV2: Decoder[Person] =
    Decoder.forProduct3("id", "name", "surname")(Person.apply)

  given Decoder[Person] = decodePersonV2.or(decodePersonV1)

The or combinator walks through the decoders. First the current one is tried, then the older one. Martin Kleppmann uses the term schema-on-read, and the data schema in the example above is the Decoder for Person. Keeping backward compatibility is a necessary aspect of working with schema-on-read, unlike traditional relational stores, which guarantee that data matches the schema by doing checks at write time.

This approach has a plus: the storage can hold data in different formats, and explicit migration is redundant.

There is a minus too: if the structure changes a lot, a duplicate Scala model may be needed. Sometimes — rewriting the data with the current Encoder, if MQL queries run on more than just the key.

Testing

To check the correctness of the data schema, the team uses round-trip — a technique. The idea is to encode a value and read it back. If it is unchanged, it works as intended. Let's look at a new example with UserId:

import cats.Eq
import ru.m2.calypso.{Decoder, Encoder}

opaque type UserId = Long
object UserId:
  def apply(value: Long): UserId = value

  given Encoder[UserId] = Encoder.given_Encoder_Long
  given Decoder[UserId] = Decoder.given_Decoder_Long
  given Eq[UserId]      = Eq.fromUniversalEquals

This uses opaque type, a concept from Scala 3. It lets you give an alias to a primitive type, which is often used in domain modeling. Avoiding primitive types reduces the number of careless mistakes.

def f(userId: Long, timestamp: Long) = ???
def g(userId: UserId, timestamp: Timestamp) = ???

In the first example the arguments can be swapped. In the second the problem is solved, and using domain types like UserId also adds expressiveness to the code.

A tricky codec description is a way around the limitations of opaque type implementation. UserId and Long inside the object UserId are identical, so the compiler is helped to avoid an infinite loop when searching for given instances. The team tried different tricks and settled on this one. The codec test code:

import munit.DisciplineSuite
import org.scalacheck.{Arbitrary, Gen}
import ru.m2.calypso.testing.CodecTests

class CodecSuite extends DisciplineSuite:
  checkAll("Codec[UserId]", CodecTests[UserId].codec)

given Arbitrary[UserId] = Arbitrary(Gen.long.map(UserId.apply))

The tests come out concise; the round-trip logic lives in CodecTests from the calypso-testing module. The example uses a property-based approach. Data samples with random values, including edge cases, are generated on every run. That is faster than listing them explicitly. The test runs several times, checking several UserId values. Arbitrary is a ScalaCheck abstraction: it states that a generator of arbitrary UserId values exists, made from a Long generator.

Cats, refined

The idea of an expressive model can be taken further with types like NonEmptyList from cats and NonEmptyString from the refined library, which expresses type constraints. For example, non-empty strings or only positive numbers. This lets you describe domain rules at the type level and also get free documentation that needs no maintenance. At M2 they like refined: it saves time and reduces the amount of code. Share in the comments how you use the Scala type system for domain modeling.

Conclusion

The design of calypso is based on ideas by Tony Morris and Mark Hibberd, who started argonaut and circe. The team has used the library for three years in applications that people use every day. Support the project on GitHub, open an Issue or Pull Request, or stop by to see how libraries are written in Scala 3. The calypso source code uses only the necessary language constructs, to stay readable.

Using ADT and refined raises the level of Domain-Driven Design implementation, and calypso simplifies working with schema-on-read and lets you implement DDD Aggregate and Transactional outbox on MongoDB clearly.

Try calypso in your Scala 3 or Scala 2 project.