Lab: StatefulSets

Introduction
StatefulSet is the workload API object used to manage stateful applications.
Manages the deployment and scaling of a set of Pods, and provides guarantees about the
ordering and uniqueness of these Pods.
Like a Deployment, a StatefulSet manages Pods that are based on an identical container spec.
Unlike a Deployment, a StatefulSet maintains a sticky identity for each of their Pods. These pods
are created from the same spec, but are not interchangeable: each has a persistent identifier
that it maintains across any rescheduling.
In this Lab, you will learn below items:
Objectives:

• Create a StatefulSet
• Manage a StatefulSet
• Delete pod in a StatefulSet
• Scale replicas in a StatefulSet
• Update StatefulSet
• Clean up

Note: Ensure you have running cluster deployed

1. Ensure that you have logged-in as root user with password as linux on kube-master node.

2. Let us clone the git repository which contains manifests required for this exercise, by
executing the below command.

# git clone https://github.com/EyesOnCloud/k8s-statefulset.git

Output:

Student Material – Do Not Re-distribute. For any queries contact:

naushad.p.tech@gmail.com or https://www.linkedin.com/in/naushadpasha/
3. Let us view the manifest file.

# cat -n ~/k8s-statefulset/statefulset.yaml
Output:

Student Material – Do Not Re-distribute. For any queries contact:

naushad.p.tech@gmail.com or https://www.linkedin.com/in/naushadpasha/
4. Let us create statefulset using the manifest file statefulset.yaml, by executing the below
command:
# kubectl apply -f ~/k8s-statefulset/statefulset.yaml
Output:

5. Using Stable Network Identities

Each Pod has a stable hostname based on its ordinal index
# for i in 0 1; do kubectl exec "web-$i" -- sh -c 'hostname';
done
Output:

6. Using nslookup on the Pods' hostnames, you can examine their in-cluster DNS addresses:

# ubectl run test2 --image busybox:1.28 -it --rm -- nslookup

web-0.nginx

Output:

# ubectl run test2 --image busybox:1.28 -it --rm -- nslookup

web-1.nginx

Student Material – Do Not Re-distribute. For any queries contact:

naushad.p.tech@gmail.com or https://www.linkedin.com/in/naushadpasha/
2. Verify the pods :
2.1 Let us verify the nginx Service, by executing the below command.
# kubectl get service nginx
Output:

2.2 Let us verify the statefulset, by executing the below command.

# kubectl get statefulset web
Output:

3. Ordered Pod Creation

For a StatefulSet with n replicas, when Pods are being deployed, they are created sequentially,
ordered from {0..n-1}. Examine the output of the kubectl get command in the first terminal.

# kubectl get pods -l app=nginx

Output:

Notice that the web-1 Pod is not launched until the web-0 Pod is Running (see Pod Phase) and
Ready (see type in Pod Conditions).
Open duplicate terminal to watch
4.4 In one terminal, watch the StatefulSet's Pods:

# kubectl get pod -w -l app=nginx

Output:

Student Material – Do Not Re-distribute. For any queries contact:

naushad.p.tech@gmail.com or https://www.linkedin.com/in/naushadpasha/
3.1 In a second terminal, use kubectl delete to delete all the Pods in the StatefulSet:
# kubectl delete pod -l app=nginx
Output:

3.2 Wait for the StatefulSet to restart them, and for both Pods to transition to Running and
Ready:
# kubectl get pod -w -l app=nginx
Output:

Student Material – Do Not Re-distribute. For any queries contact:

naushad.p.tech@gmail.com or https://www.linkedin.com/in/naushadpasha/
3.3 Use kubectl exec and kubectl run to view the Pods' hostnames and in-cluster DNS entries.
First, view the Pods' hostnames:
# for i in 0 1; do kubectl exec web-$i -- sh -c 'hostname';
done
Output:

3.4 Let us start a new shell.

# kubectl run -i --tty --image busybox:1.28 dns-test --

restart=Never --rm /bin/sh

3.5 In that new shell, run: to notice a new pod ip has been assigned after deletetion and
recreation of statefulset.

# nslookup web-0.nginx
# nslookup web-1.nginx
Output:

exit the container shell: exit

Student Material – Do Not Re-distribute. For any queries contact:

naushad.p.tech@gmail.com or https://www.linkedin.com/in/naushadpasha/
4. Writing to Stable Storage:
Get the Persistent Volume Claims for web-0 and web-1:

# kubectl get pvc -l app=nginx

Output:

The StatefulSet controller created two Persistent Volume Claims that are bound to two
Persistent Volumes.
As the cluster used in this lab is configured to dynamically provision Persistent Volumes, the
Persistent Volumes were created and bound automatically.
4.1 Write the Pods' hostnames to their index.html files and verify that the NGINX webservers
serve the hostnames:
# for i in 0 1; do kubectl exec "web-$i" -- sh -c 'echo
"$(hostname)" > /usr/share/nginx/html/index.html'; done
# for i in 0 1; do kubectl exec -i -t "web-$i" -- curl
http://localhost/; done
Output:

Note: If you instead see 403 Forbidden responses for the above curl command, you will need to
fix the permissions of the directory mounted by the volume Mounts (due to a bug when using
hostPath volumes), by running:
(for i in 0 1; do kubectl exec web-$i -- chmod 755 /usr/share/nginx/html; done)
before retrying the curl command above.

Student Material – Do Not Re-distribute. For any queries contact:

naushad.p.tech@gmail.com or https://www.linkedin.com/in/naushadpasha/
4.2 In one terminal, watch the StatefulSet's Pods:
# kubectl get pod -w -l app=nginx
Output:

4.3 In a second terminal, delete all of the StatefulSet's Pods:

# kubectl delete pod -l app=nginx
Output:

Student Material – Do Not Re-distribute. For any queries contact:

naushad.p.tech@gmail.com or https://www.linkedin.com/in/naushadpasha/
4.4 Verify the output of the kubectl get command in the first terminal, and wait for all of the
Pods to transition to Running and Ready.
# kubectl get pod -w -l app=nginx
Output:

4.5 Verify the web servers continue to serve their hostnames:

# for i in 0 1; do kubectl exec -i -t "web-$i" -- curl

http://localhost/; done
Output:

Even though web-0 and web-1 were rescheduled, they continue to serve their hostnames
because the PersistentVolumes associated with their PersistentVolumeClaims are remounted to
their volume Mounts. No matter what node web-0 and web-1 is scheduled on, their
PersistentVolumes will be mounted to the appropriate mount points.

Student Material – Do Not Re-distribute. For any queries contact:

naushad.p.tech@gmail.com or https://www.linkedin.com/in/naushadpasha/
5. Scaling a StatefulSet:
Scaling a StatefulSet refers to increasing or decreasing the number of replicas. This is
accomplished by updating the replicas field. You can use either kubectl scale or kubectl patch
to scale a StatefulSet.
5.1 Scaling Up:
In one terminal window, watch the Pods in the StatefulSet:
# kubectl get pod -w -l app=nginx
Output:

In another terminal window, use kubectl scale to scale the number of replicas to 5:

# kubectl scale sts web --replicas=5

Output:

Student Material – Do Not Re-distribute. For any queries contact:

naushad.p.tech@gmail.com or https://www.linkedin.com/in/naushadpasha/
Examine the output of the kubectl get command in the first terminal, and wait for the three
additional Pods to transition to Running and Ready.
# kubectl get pods -w -l app=nginx
Output:

5.2 Scaling down:

In one terminal, watch the StatefulSet's Pods:

# kubectl get pods -w -l app=nginx

Output:

In another terminal, use kubectl patch to scale the StatefulSet back down to three replicas:
# kubectl patch sts web -p '{"spec":{"replicas":3}}'
Output:

Student Material – Do Not Re-distribute. For any queries contact:

naushad.p.tech@gmail.com or https://www.linkedin.com/in/naushadpasha/
Wait for web-4 and web-3 to transition to Terminating.
# kubectl get pods -w -l app=nginx
Output:

6. Let us cleanup by deleting, by executing the below command.

# kubectl delete -f ~/k8s-statefulset/
Output:

Student Material – Do Not Re-distribute. For any queries contact:

naushad.p.tech@gmail.com or https://www.linkedin.com/in/naushadpasha/