In Kubernetes, a StatefulSet is a replicated group of Pods, similar to a ReplicaSet.
Each replica gets a stable host name with a unique index (for example, database-0, database-1, and so on).
Each replica is created from the lowest index to the highest, and creation is blocked until the pod with the previous index is running and available. This also applies to scaling.
When a StatefulSet is deleted, each of the managed replica pods is also deleted in descending order. The same applies to scaling down.
It turns out that this simple set of requirements makes deploying stateful applications in Kubernetes much easier. For example, combining stable host names (like database-0) with ordering constraints means that every replica except the first can reliably refer to database-0 for discovery and establishing the replication quorum.
Today we will deploy a replicated MongoDB cluster with a StatefulSet.
Start by creating a replicated set of three MongoDB modules using the StatefulSet object. The main application container uses the mongo:3.4.24 container image and runs the mongod process. When you start mongod, the MongoDB process starts and it runs in the background. The process has several default settings, such as storing data in /data/db and running on port 27017.
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: mongo
spec:
serviceName: "mongo"
replicas: 3
selector:
matchLabels:
app: mongo
template:
metadata:
labels:
app: mongo
spec:
containers:
- name: mongodb
image: mongo:3.4.24
command:
- mongod
- --replSet
- rs0
ports:
- containerPort: 27017
name: peer
Do not create the object yet, since we will be making further changes to the YAML manifest.
Initializing the MongoDB cluster with an init container
To automate the deployment of our StatefulSet-based MongoDB cluster, we are going to add an extra container to the pods to perform initialization.
To configure this module without building a new Docker image, we are going to use a ConfigMap to add a script to the existing MongoDB image.
We are going to run this script with an init container. Init containers (or init Containers) are specialized containers that run once when a pod starts. They are typically used when there is a small amount of setup work that is worth doing before the main application starts. A pod definition has a separate initContainers list where you can define init containers.
The final step in building our MongoDB cluster is adding liveness checks to our Mongo-serving containers:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: mongo
spec:
serviceName: "mongo"
replicas: 3
selector:
matchLabels:
app: mongo
template:
metadata:
labels:
app: mongo
spec:
containers:
- name: mongodb
image: mongo:3.4.24
command:
- mongod
- --replSet
- rs0
ports:
- containerPort: 27017
name: web
livenessProbe:
exec:
command:
- /usr/bin/mongo
- --eval
- db.serverStatus()
initialDelaySeconds: 10
timeoutSeconds: 10
# This container initializes the mongodb server, then sleeps.
- name: init-mongo
image: mongo:3.4.24
command:
- bash
- /config/init.sh
volumeMounts:
- name: config
mountPath: /config
volumes:
- name: config
configMap:
name: "mongo-init"
Replace the contents of the mongo.yaml file. On the next step we will create a ConfigMap named mongo-init.
Creating the ConfigMap
Note that the Pod mounts a ConfigMap volume named mongo-init. This ConfigMap contains the script that runs our initialization. First, the script checks whether it is running on mongo-0. If it is on mongo-0, it creates the ReplicaSet with the same command we ran imperatively earlier. If it is on another Mongo replica, it waits until the ReplicaSet exists (the correct phrasing would be: until it appears), and then registers itself as a member of that ReplicaSet.
The following YAML manifest contains the complete ConfigMap object:
apiVersion: v1
kind: ConfigMap
metadata:
name: mongo-init
data:
init.sh: |
#!/bin/bash
# Need to wait for the readiness health check to pass so that the
# mongo names resolve. This is kind of wonky.
until ping -c 1 ${HOSTNAME}.mongo; do
echo "waiting for DNS (${HOSTNAME}.mongo)..."
sleep 2
done
until /usr/bin/mongo --eval 'printjson(db.serverStatus())'; do
echo "connecting to local mongo..."
sleep 2
done
echo "connected to local."
HOST=mongo-0.mongo:27017
until /usr/bin/mongo --host=${HOST} --eval 'printjson(db.serverStatus())'; do
echo "connecting to remote mongo..."
sleep 2
done
echo "connected to remote."
if [[ "${HOSTNAME}" != 'mongo-0' ]]; then
until /usr/bin/mongo --host=${HOST} --eval="printjson(rs.status())" \
| grep -v "no replset config has been received"; do
echo "waiting for replication set initialization"
sleep 2
done
echo "adding self to mongo-0"
/usr/bin/mongo --host=${HOST} \
--eval="printjson(rs.add('${HOSTNAME}.mongo'))"
fi
if [[ "${HOSTNAME}" == 'mongo-0' ]]; then
echo "initializing replica set"
/usr/bin/mongo --eval="printjson(rs.initiate(\
{'_id': 'rs0', 'members': [{'_id': 0, \
'host': 'mongo-0.mongo:27017'}]}))"
fi
echo "initialized"
You will notice that the script defined in the ConfigMap exits immediately. This matters when using initContainers. Each init container waits for the previous container to finish before it runs. The main application container waits for all init containers to complete. If this script never exited, the main mongo server would never start.
Create the ConfigMap object first:
kubectl apply -f mongo-configmap.yaml
Next we will create the StatefulSet object, which can now reference the ConfigMap by name.
kubectl apply -f mongo.yaml
Once the objects exist, the differences between a ReplicaSet and a StatefulSet become obvious. Since creating the pods of a StatefulSet takes a while, you will have to re-run the same command several times until the count reaches three replicas:
kubectl get pods
There are two important differences between this and what you would see with a ReplicaSet. First, each replicated Pod has a numeric index (0, 1, …) instead of the random suffix that the ReplicaSet controller adds. Second, the pods are created slowly in order, rather than all at once the way a ReplicaSet would do it.
Exposing MongoDB as a service
After creating the StatefulSet we also need to create a "headless" service to manage the DNS records for the StatefulSet. In Kubernetes a service is called "headless" if it has no virtual cluster IP address. Since with StatefulSets every Pod has a unique identifier, a load-balancing IP address for a replicated service does not really make sense. You can create a headless service by using clusterIP: None in the service spec:
apiVersion: v1
kind: Service
metadata:
name: mongo
spec:
ports:
- port: 27017
name: peer
clusterIP: None
selector:
app: mongo
After this service is created, four DNS records are normally populated. As usual, mongo.default.svc.cluster.local is created, but unlike a standard service, a DNS lookup on that host name returns all the addresses in the StatefulSet. In addition, records are created for mongo-0.mongo.default.svc.cluster.local, as well as for mongo-1.mongo and mongo-2.mongo. Each of them resolves to a specific replica-index IP address in the StatefulSet. So with StatefulSets you get clearly defined stable names for every replica in the set. This is often very useful when configuring a replicated storage solution.
Create the service with the following command:
kubectl apply -f mongo-service.yaml
Once we have combined StatefulSets, pvc, and liveness checks, we have a secure, scalable MongoDB deployment running in Kubernetes. Although this topic covered MongoDB, the steps for creating a StatefulSet to manage other storage solutions are very similar, and the same patterns can be followed.