MongoDB has been moving toward the strictness of the relational model (an optional data schema, join queries, aggregations, and so on), but it is still a document database and still expects that documents of arbitrary structure can be stored. When MongoDB is used from dynamically typed languages (JavaScript, Python), storing or generating objects causes no trouble, because you do not need to define the structure of the object you are reading or writing in advance. But what do you do when using MongoDB drivers from statically typed languages — for example, the KMongo library for Kotlin?
This article covers techniques for working with unstructured data that keep the advantages of serialization (Jackson or kotlinx.serialization) alongside reflection-based mechanisms for fetching arbitrary documents.
A few words about the KMongo library itself. This open source library is built on the Core MongoDB Java Driver and provides convenient syntax for running queries and iterating over databases, collections, and query results using Kotlin features (in particular, infix operators let you write queries like Jedi::age lt yoda?.age). It also supports implicit serialization when reading documents from Mongo and when saving new documents. KMongo additionally lets you register adapters for non-blocking operation execution (using Rx or coroutines).
To install KMongo, add the core library and the non-blocking execution adapter to build.gradle — for example, to support coroutines and kotlinx.serialization:
dependencies {
implementation("org.litote.kmongo:kmongo-serialization:4.8.0")
implementation("org.litote.kmongo:kmongo-coroutine-serialization:4.8.0")
}
To create a connection to MongoDB, call the createClient function on the KMongo client. With non-blocking execution support installed, the result exposes a coroutine getter for obtaining an asynchronous version of the connection, like this:
val client = KMongo.createClient(connectionString = "mongodb://mongoip:27017").coroutine
The resulting CoroutineClient lets you list the available databases (listDatabases), connect to a database (getDatabase(name)), and subscribe to a change stream (watch). For example, to connect to the test database you can use the following call:
val database = client.getDatabase("test")
If the client was obtained as a CoroutineClient, the database is also available through the CoroutineDatabase wrapper. You can then run queries against the connected database — and this is where we first run into the need to work with structured objects. The structure can be any data class; it is used when executing queries and when defining selection conditions. For example, to select all messages where the numeric field type equals 1:
import org.litote.kmongo.*
import kotlinx.serialization.*
@Serializable
data class TestItem(val type:Int, val message:String)
suspend fun getItems(database:CoroutineDatabase):List<TestItem> {
val collection = database.getCollection<TestItem>("items")
return collection.find(TestItem::type eq 1)
}
New records can be added to a collection in the same way:
collection.insertOne(TestItem(1, "Test"))
The trouble starts when you need to insert or fetch data with complex types or arbitrary structure. For the first scenario you can create codecs that interpret arbitrary (including binary) Mongo data types as Kotlin objects. Dates are supported out of the box; for anything else you can register a codec through ObjectMappingConfiguration.addCustomCodec for the corresponding result type. The second scenario is where a real difficulty appears, because there is no serialization mechanism for dynamic structures. Suppose the items collection also stores an extra meta field holding a map with string keys and values of arbitrary types. The obvious solution might look like this definition:
data class TestItem(val type:Int, val message:String, val meta:Map<String,An
The difficulty here is that dynamic data types are not supported when decoding the Mongo response. What do you do in this situation? You can use an object of the BsonDocument class, which lets you work with individual named fields of a Mongo object — including listing all fields present in a document and determining their types. BsonDocument can be used both when executing a query and inside a serializable object. Try fetching a data structure with an unstructured field:
suspend fun getData(db:CoroutineDatabaes):List<TestItem> {
val items = mutableListOf<TestItem>()
val collection = db.getCollection<BsonDocument>("items")
collection.consumeEach { item ->
val metaData = mutableMapOf<String,Any>()
val type = item.getInt32("type").value
val message = item.getString("message").value
val meta = item["meta"]?.asDocument()
meta?.keys?.forEach { key ->
metaData[key] = meta.getValue(key).value
}
items.add(TestItem(type, message, metaData))
}
return items
}
Here we use the ability to iterate over the key set of a BsonDocument and the common interface for getting the value of any field (a BsonDocument may also come back, in which case it can be handled recursively). An object with an arbitrary structure can be stored in the same way:
suspend fun addData(db:CoroutineDatabase, data:TestItem) {
val meta = BsonDocument()
data.meta.keys.forEach { key ->
val value = data.meta[key]
if (value is String) {
meta[key] = BsonString(value))
} else if (value is Boolean) {
meta[key] = BsonBoolean(value))
} else if (value is Double) {
meta[it] = BsonDouble(value)
} else if (value is Int) {
meta[it] = BsonInt64(value)
}
}
val doc = BsonDocument(
listOf(
BsonElement("type", BsonInt32(data.type)),
BsonElement("message", BsonString(data.message)),
BsonElement("meta", meta),
)
)
db.getCollection<BsonDocument>("items").insertOne(doc)
}
So documents of arbitrary structure can be stored in and fetched from MongoDB in Java/Kotlin applications. Writing custom serializers for kotlinx.serialization / Jackson and custom codecs for the driver is, of course, an option, but in most cases direct access to BsonDocument turns out to be enough for working with complex documents.