In the spring of 2021, a dramatic event took place in Strasbourg, France: one of Europe's largest hosting providers (OVH) burned down completely. In just a few hours, the fire cut off access to a million popular websites and online services around the world. One of the probable causes was human error. As a result, not only the data center itself but the provider's entire business was put at risk. Unfortunately, a fire is not the only problem with big data. No less dangerous are high-load systems: for example, an application can no longer cope with a sudden load spike, the whole infrastructure runs at its limit, and there is no headroom left for growth. To jump ahead — every one of the problems listed has a solution. But first things first.
What to do when DBMS volumes and load keep growing and we have already run out of options
An important aspect of all this is the design and operation of high-load information systems — the bottlenecks in the architecture of such software solutions.
Traditionally, the biggest bottleneck in any system's architecture is the database management system (DBMS). You can optimize application software as much as you like, but you will still hit the limits of DBMS query performance. We have already talked about dropping traditional DBMSs in favor of NoSQL, sacrificing strict consistency but finding a compromise — eventual consistency. But what do you do when DBMS volumes and load keep growing and you have exhausted every option for vertical scaling?
The optimal solution is probably sharding. This approach splits a database, or individual database objects, into independent segments, each managed by a separate database server instance, usually placed on its own compute node.
Unlike partitioning, which stores pieces of database objects separately under a single DBMS instance, segmentation lets you use distributed computing techniques. It is more complex to implement, however, since it requires coordinating many instances — and their interaction has to treat the entire set of segments as a single database.
Segmentation is widely used in NoSQL DBMSs (such as Cassandra, Couchbase, MongoDB), analytical DBMSs (Teradata Database, Netezza, Greenplum), and search engines (Elasticsearch, Solr). It is also implemented in some traditional relational DBMSs (the Sharding option in Oracle Database). For DBMSs that do not support segmentation, the application routes queries to several DBMS instances itself.
To make this clearer, let's look at an architecture that uses a sharded DBMS, with MongoDB as the example:
Let's break down each element of the architecture:
Application — the application itself. Take a typical microservice application for simplicity; its architecture is out of scope for this article. What matters is that the application talks to the DBMS;
Mongos — the query router from the application to a specific DBMS node. There can be any number of routers; it depends on the load profile and the specifics of the application;
Config Servers — the DBMS configuration server (a cluster). It stores the metadata of the sharded DBMS cluster. It is a typical MongoDB cluster in a ReplicaSet configuration;
Shard — a shard: a DBMS cluster responsible for storing a data segment. It is a typical MongoDB cluster in a ReplicaSet configuration. The number of shards can be anything; it depends on the load profile, data volumes, and performance and fault-tolerance requirements. In effect, each shard is a separate DBMS instance.
How do you choose a shard key?
Horizontal scaling implies distributing a dataset and the load across several DBMS nodes. But how does it work? That is what the so-called shard key is for. The mechanism: entities tied to the same shard key value are grouped into a dataset by that key. This dataset is stored within a single physical shard, which makes processing the data much easier.
So if the shard key of an object is known, you can always answer the questions:
- Where should the data be stored?
- Where can the requested data be found?
That raises a fair question — how do you choose a shard key? I'll note that a good key has several properties: cardinality and immutability, and the second is used everywhere in frequent queries.
The natural choice of shard key is the entity's identifier. If you have one, use it! Ideally, this identifier belongs to the UUID family (universally unique identifier).
Essentially, a UUID is an identification standard used in software development.
The main purpose of a UUID is to let distributed systems uniquely identify information without a central coordinating point. So anyone can generate a UUID and use it to identify something, with a reasonable level of confidence that this identifier will never accidentally be used for something else.
Another important point: the right shard key keeps data chunks (chunks) evenly distributed across the cluster's shards, which means high read performance. Also, the more evenly data is distributed across shards, the less load on the balancer, whose job is exactly to even things out. And I hope everyone remembers perfectly well that the chunk balancing process is far from free!
MongoDB offers two key types out of the box:
- Ranged — splitting data into contiguous ranges;
- Hashed — splitting data based on a hash function.
The choice is yours, but in most cases I would recommend hashed distribution (hashed shard key). The reason is simple: a hash function can turn even a key that is poor by formal criteria into a good one. To put it even more simply, a hash function's job is to distribute entities evenly across shards. I could go deep into serious mathematics and talk about Pearson's criterion and other interesting things, but there is no need: the engineers at MongoDB Inc. have already thought all of this through for us and picked a good hash function for the sharding task. All that is left for us is to use it and enjoy it.
"Games" of the mind: how to avoid "parasitic" traffic in the data center?
We're done with keys, and now we move on to the "flammable" terms: georedundancy and disaster tolerance.
Disaster tolerance usually means a georedundancy system: the information system's infrastructure is located in several data centers (NOC). Disaster tolerance keeps the business processes implemented in the information system running when an entire data center fails. It is hard to believe, but even now there are large-scale power grid failures, and they last a long time. And even the complete burnout of a data center.
Disaster-tolerant data center solutions fall into 2 classes: an active-passive configuration, where the load runs on a single site, and a metro cluster configuration, where the load runs on both sites at the same time.
Let's look at an example of a disaster-tolerant solution built on a MongoDB DBMS in an active-passive configuration, with the DBMS topology spread across two data centers (NOC). Let's figure out how it works!
In effect, we split each shard into two parts — one part runs in NOC1, the other in NOC2. The active data center holds the shard's PRIMARY node and half of the secondary nodes. Data replication toward NOC2 runs from one of the secondary DBMS nodes in the active data center (NOC1), not from the PRIMARY as we are used to. Note that this database node is marked hidden. Hidden from whom? From the application, of course — otherwise extra read load would fall on that node.
Within the data center itself, cascading replication is configured. Why? The thing is, there is a communication link between NOC1 and NOC2, and since they are geographically separated, the link has a latency of several tens of milliseconds. If replication ran from the main (PRIMARY) node of the active shard in NOC1, it would slow down the DBMS cluster in that data center. The point is, when writing, the PRIMARY node waits for confirmation from a quorum of replicas (SECONDARY) that they have finished the write, and since there is latency on the link, confirmations from replicas located in NOC2 would arrive late too. On top of that, we would simply clog the link between the data centers with "parasitic" replication traffic. So replication into NOC2 runs from a dedicated node in that same data center, which in turn replicates from a source located in NOC1. Phew, that probably confused everyone… To get a clear picture of what is going on, I suggest reading this paragraph a few times.
So we're done with shard replication. The config server is even simpler — it is a single distributed ReplicaSet spanning both data centers.
Next come the query routers — Mongos. Those are simple too. Like the application itself, they are simply duplicated in both data centers. There is no interaction between these components; they work completely independently of each other.
What happens when disaster strikes NOC1, when it suddenly turns to ash or a "pumpkin"? The procedure is as follows:
- Remove the nodes located in NOC1 from the replica sets for the shards and config servers
- Change the role of the main node — the replication source — in NOC2 to PRIMARY
- Change the role of one of the secondary nodes in the config server replica set in NOC2 to PRIMARY
- Remove the hidden flag from all nodes in NOC2
- Everything works, the data is safe. True, this did require an engineer's intervention, and after such a hectic round of running around, they will surely demand a day off at the company's expense.
Even sharding has drawbacks
Of course, any wonderful thing in the world, including a sharded DBMS, has drawbacks worth keeping in mind:
- First, a sharded cluster is expensive! Admit it: a cluster of 3 physical servers costs less than a cluster of 30 nodes. Let's do the math: a regular (unsharded) DBMS cluster, i.e. a typical ReplicaSet, is a primary plus a pair of secondaries — 3 nodes in total. A sharded cluster is at least 2 shards, usually more (three nodes each), plus a replica set for the config servers and servers for mongos.
- Second, the count operation does not always work on sharded collections. It can return more documents than there actually are. The reason is the balancer, which "pours" documents from one shard to another in the background. At some point, documents have already been written to the target shard but not yet deleted from the source — count will count them twice. Note that with a specific query, the count({query}) operation works correctly. Newer DBMS versions also provide a way to get the total number of documents in a collection with estimatedDocumentsCount(). Read more about this on the vendor's site.
- Third, a sharded cluster is harder to administer. You have to ensure reliable network connectivity between all DBMS nodes, put every DBMS host under monitoring and alerting, arrange regular backups for each shard, keep the cluster highly available, and so on. All of this compounds the complexity of operating and administering the DBMS and the information system as a whole, compared with a regular clustered DBMS configuration (a single replica set).
Should sharding be dismissed for its cost and administrative complexity? The question is rhetorical: what is at stake is not so much money and time as your company's reputation and growth.