ServiceAccounts and SCCs - Running pods with more permission's

Jan 17, 2022

In this blog we will take a look at SCC permissions that are needed to run a pod
with escalated permissions and how to use a serviceaccount (sa) to run such a pod
or deployment.

Small thing: I use the term pod and container in this post but I’m referring to the
same idea. In Openshift and Kubernetes we talk about pods being the smallest unit
of measurement. Each pod usually runs one container but there are pods that run
multiple containers.

Understanding Rootless
I won’t explain fully why and what rootless means in general but will focus on the
parts that are importent to know in a Openshift environment and according to the
EX280 exam objectives. If you do wanna go deep on rootless, namespaces and more I
highly recommend the following video’s by RedHat:

Overview of Rootless Podman: Part 1 - Understanding Root Inside and Outside a

Container
Overview of Rootless Podman: Part 2 - How User Namespaces Work in Rootless
Containers
UIDs and Namespaces
So what are Namespaces? A namespace is a feature of the Linux kernel that allows
you to segment resources 1. All things that make up a running system or process are
at home in a Namespace. You’ll find Process ID’s (PID) in there as well as mounts
and user ID’s (UID). Simply said, a Namespace is a organizational container to
partition resources in. Now, your Linux base system has a great way of
orchestrating all these different resources based on their unique ID’s and maps
process ID’s that are spawned inside a container back to the host system so that
the stuff that needs running (the processes that make up the containers) can
actually get resources and stay separated from each other.

So, when we are talking about container technology and containers being “contained”
we are actually talking about this awesome namespace feature.

So, what are the risks?

There are different risks associated with running containers that default to a root
account. The obvious risk is the same as on normal Linux server. Imagine all
accounts that are available on a Linux host having root privileges. Now imagine
someone gaining access to that system. When you run a pod that defaults to the root
account (aka a root container) you’re basically exposing the same risk. You can
have all the security of a cluster, but if someone manages to drop a executable in
a pod or gain access to it they could do a lot of damage.

If you want to go deeper on rootless containers, the risks and how to avoid them,
check out this video by Synk.

Openshift to the rescue(?)

One of the things that Openshift takes care of (in comparison with the out-of-the-
box Kubernetes cluster) is that it won’t allow root containers to run by default.
This can lead to some issues as some containers need more access to the underling
host or special resources or that the images are not setup for rootless usage. For
these cases Openshift provides Security Context Constraints to manage elevated
rights for pods.

SCC’s and SA’s

SCC? SA? What do you mean? Let me (try) to explain:
SCC: Security Context Constrains. A SCC is used to provide access to a certain
resource like: anyuid (which allows the pod to run under any UID), hostaccess
(which gives the pod host access) and hostnetwork (which, well I’ll let you guess
that one yourself)

To see all SCCs that are available in you cluster you can use:

$ oc get scc -o name

securitycontextconstraints.security.openshift.io/anyuid
securitycontextconstraints.security.openshift.io/hostaccess
securitycontextconstraints.security.openshift.io/hostmount-anyuid
securitycontextconstraints.security.openshift.io/hostnetwork
securitycontextconstraints.security.openshift.io/machine-api-termination-handler
securitycontextconstraints.security.openshift.io/nonroot
securitycontextconstraints.security.openshift.io/privileged
securitycontextconstraints.security.openshift.io/restricted
(to see more details, use oc get scc)

SA: A Service Account is used to run a pod with a special SCC. In Openshift you
create Service Accounts and bind them to a deployment to hand out permissions. You
can see

Running a container with elevated permissions

A great container image to use as an example for this post is the nginx container
which is available in both a root a rootless version. For our example we will
create two apps based on the two different images. One is the official nginx
container image and the second one is the bitnami/nginx rootless version:

Creating the deployments

We will create a new namespace and two apps:

$ oc new-project root-more-or-less
$ oc new-app --name root-container --image docker.io/nginx
$ oc new-app --name rootless-container --image docker.io/bitnami/nginx
Strait away we can see some issues with our root pod:

$ oc get pods
NAME READY STATUS RESTARTS AGE
root-container-67c49b777d-wdzn5 0/1 Error 3 (28s ago) 58s
rootless-container-59c496f955-zh7qd 1/1 Running 0 76s
Troubleshooting
Let’s have a look at whats going on here. We will start with a simple oc status
command:

$ oc status --suggest
....
Errors:
* pod/root-container-67c49b777d-wdzn5 is crash-looping

The container is starting and exiting repeatedly. This usually means the container
is unable to start, misconfigured, or limited by security restrictions. Check the
container logs with

oc logs root-container-67c49b777d-wdzn5 -c root-container

Current security policy prevents your containers from being run as the root user.
Some images may fail expecting to be able to change ownership or permissions on
directories. Your admin can grant you access to run containers that need to run as
the root user with this command:
 oc adm policy add-scc-to-user anyuid -n root-or-less -z default
....
Of the bat we get two hints of fixing this. oc status is telling us to check the
logs of the pod and that the pod might not be able to run due to a SCC. Let’s not
blindly follow the suggestion of adding anyuid to this pod and explore further with
the logs:

$ oc logs root-container-67c49b777d-wdzn5
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform
configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-
default.sh
10-listen-on-ipv6-by-default.sh: info: can not modify
/etc/nginx/conf.d/default.conf (read-only file system?)
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2022/01/22 15:51:47 [warn] 1#1: the "user" directive makes sense only if the master
process runs with super-user privileges, ignored in /etc/nginx/nginx.conf:2
nginx: [warn] the "user" directive makes sense only if the master process runs with
super-user privileges, ignored in /etc/nginx/nginx.conf:2
2022/01/22 15:51:47 [emerg] 1#1: mkdir() "/var/cache/nginx/client_temp" failed (13:
Permission denied)
nginx: [emerg] mkdir() "/var/cache/nginx/client_temp" failed (13: Permission
denied)
As we can see, the pod is expecting to be run as a root user:

[warn] 1#1: the "user" directive makes sense only if the master process runs with
super-user privileges, ignored in /etc/nginx/nginx.conf:2
Before we fix this, let’s check out the SCC’s the pod’s are currently using:

$ oc describe pod | grep -i SCC

openshift.io/scc: restricted
openshift.io/scc: restricted
Now lets use this amazing policy command to find out what SCC our root container
wants to user:

$ oc get pod root-container-67c49b777d-wdzn5 -o yaml \

| oc adm policy scc-subject-review -f -
RESOURCE ALLOWED BY
Pod/root-container-67c49b777d-wdzn5 anyuid
Looks like our pod wants to use anyuid just as suggested when we ran the oc status
command.

Creating a ServiceAccount
As mentioned before, if you want to run a pod with a different SCC you’ll need to
create a ServiceAccount to run the pod. The SA will provide the runtime with the
elevated permissions that we add to the SA

We will create a SA called ok-go-for-it in our namespace with:

$ oc create sa ok-go-for-it \
--namespace root-more-or-less
serviceaccount/ok-go-for-it created
A small tip, always create the SA with a --namespace (or -n) flag to ensure the SA
is bound to the namespace.
Giving special permissions to a ServiceAccount
Now let’s add some permissions to our brand spanking new SA. We use oc adm policy
to add permissions to users, groups and ServiceAccounts. When we add permissions to
a SA we use the -z flag:

$ oc adm policy add-scc-to-user -h

Add a security context constraint to users or a service account.

Usage:
oc adm policy add-scc-to-user SCC (USER | -z SERVICEACCOUNT) [USER ...] [flags
$ oc adm policy add-scc-to-user anyuid \
-z ok-go-for-it \
--namespace root-more-or-less
clusterrole.rbac.authorization.k8s.io/system:openshift:scc:anyuid added: "ok-go-
for-it"
Adding a ServiceAccount to a Deployment
Now to update our app. We got our SA ready and our permissions permitted. Let’s not
waist time with editing yaml and set the serviceaccount for our app using oc set:

$ oc set serviceaccount deployment root-container ok-go-for-it

deployment.apps/root-container serviceaccount updated
And just to verify:

$ oc get pods
NAME READY STATUS RESTARTS AGE
root-container-5d6c7cc66b-sqhhs 1/1 Running 0 19s
rootless-container-59c496f955-zh7qd 1/1 Running 0 20m
$ oc describe pod | grep -i SCC
openshift.io/scc: anyuid
openshift.io/scc: restricted
Wrapping up
As you can see it’s pretty easy to use SAs to apply special permissions to
deployments. That doesn’t mean you should do it without thought. There are use
cases in which it’s necessary that a deployment might need to run with elevated
permissions but there are a lot of other solution’s than simply handing out the
permissions. So stay sharp!