Certified Kubernetes Administrator

(CKA)
Behrad Eslamifar
b.eslamifar@gmail.com
Security
12%
Outlines

● Know how to configure authentication and authorization.

● Understand Kubernetes security primitives.
● Know to configure network policies.
● Create and manage TLS certificates for cluster components
● Work with image securely.
● Define security contexts.
● Secure persistent key value store.
Certification Authority (CA)
Certificate-Based Authentication
TLS Handshake
TLS/SSL Workshop
 Kubernetes Certificates
root@master:~# cd /etc/kubernetes/pki # openssl x509 -in apiserver.crt -noout -text
root@master:/etc/kubernetes/pki# ls ...
apiserver.crt Validity
apiserver.key Not Before: Aug 12 10:40:17 2020 GMT
ca.crt Not After : Aug 12 10:40:17 2021 GMT
front-proxy-ca.crt Subject: CN = kube-apiserver
front-proxy-client.key ...
... X509v3 Subject Alternative Name:
DNS:ubuntu, DNS:kubernetes,
DNS:kubernetes.default, DNS:kubernetes.default.svc,
DNS:kubernetes.default.svc.cluster.local, IP
Address:10.96.0.1, IP Address:192.168.13.200
...

# openssl x509 -in ca.crt -noout -text

# openssl verify -verbose -CAfile ca.crt apiserver.crt

apiserver.crt: OK
 Renew apiserver Certificate
# cat apiserver.conf # openssl genrsa -out apiserver-new.key 2048 (optional)
[ req ]
default_bits = 2048
prompt = no # openssl req -new -key domain.key -out apiserver.csr -
default_md = sha256
req_extensions = req_ext
config apiserver.conf
distinguished_name = dn
# openssl x509 -req -in apiserver.csr -CA ca.crt -CAkey
[ dn ]
CN = kube-apiserver
ca.key -CAcreateserial -out apiserver-new.crt -days 365 -
extensions v3_ext -extfile apiserver.conf
[ req_ext ]
subjectAltName = @alt_names

[ alt_names ]
DNS.1=kubernetes
DNS.2=kubernetes.default
DNS.3=kubernetes.default.svc
DNS.4=kubernetes.default.svc.cluster
DNS.5=kubernetes.default.svc.cluster.local
DNS.6=kube.server.roo.cloud
DNS.7=ubuntu
IP.1=192.168.13.200
IP.2=10.96.0.1

[ v3_ext ]
authorityKeyIdentifier=keyid,issuer:always
basicConstraints=CA:FALSE
keyUsage=keyEncipherment,dataEncipherment
extendedKeyUsage=serverAuth
subjectAltName=@alt_names
 Admin Certificate
# cat /etc/kubernetes/admin.conf # echo <client-certificate-data> | base64 -d | openssl
... x509 -noout -text -in - > admin.crt
users:
- name: kubernetes-admin # echo <client-key-data> | base64 -d | openssl x509 -
user: noout -text -in - > admin.key
client-certificate-data:
LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM4a
kNDQWRxZ0F3SUJBZ0lJUWxPMXhxangvUzh3RFFZSktvWk
lodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVml
aWEp1WlhSbGN6QWVGUkg0Tm9JdHZJOE45cz0KLS0tLS1F
TkQgQ0VSVElGSUNBVEUtLS0tLQo=
client-key-data:
LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNS
UlFcEFJQkFBS0NBUUVBcjdRMEJMemY1dXpY3BZeWh4WXk
4dQo1eHJxOUFKRU9qcC9KQTNiSWtqUi9nN25rZzUwUHMv
YWFqMnVDNTZiV0ZlemQwOFVvUVJkcVE9PQotLS0tLUVOR
CBSU0EgUFJJVkFURSBLRVktLS0tLQo=
...
 Renew Admin Certificate
# cat admin-ssl.conf # openssl req -new -key admin.key -out admin.csr \
[ req ] -config admin-ssl.conf
default_bits = 2048
prompt = no # openssl x509 -req -in admin.csr -CA \
default_md = sha256 /etc/kubernetes/pki/ca.crt \
distinguished_name = dn -CAkey /etc/kubrenetes/pki/ca.key -CAcreateserial \
-out admin-new.crt -days 365 -extensions \
[ dn ] v3_ext -extfile admin-ssl.conf
O=system:masters
CN=kubernetes-admin

[ v3_ext ]
authorityKeyIdentifier=keyid,issuer:always
basicConstraints=CA:FALSE
keyUsage=keyEncipherment,dataEncipherment
extendedKeyUsage=clientAuth

# cat admin-new.crt | base64

Authentication and
Autherization
User Authentication: Who are you?

● Authentication Modules
○ Basic HTTP Auth
○ Static Bearer Tokens
○ X509 Client Certs
○ Service Account Tokens
○ OpenID Connect
○ Custome Webhook
Basic HTTP Auth

● Static Password File

○ API Server option
●
--basic-auth-file=SOMEFILE
password1,user1,uid1,"group1,group2,group3"
password2,user2,uid2,"group1,group2,group3"
Static Bearer Tokens

● Static Token File

○ API Server option
●
--token-auth-file=SOMEFILE
token1,user1,uid1,"group1,group2,group3"
token2,user2,uid2,"group1,group2,group3"
X509 Client Certs

● X509 Client Certs

○ API Server option
●
--client-ca-file=SOMEFILE
○ The common name (CN) as the username
○ The organization fields (O) as the groups

openssl req -new -key server.key -out server.csr -config csr.conf

Service Account Tokens

● Service Account Tokens

○ API Server option
●
--service-account-key-file
○ To use in-cluster process talk to API server
○ To use outside the cluster
○ Authenticated with
●
system:serviceaccount:(NAMESPACE):(SERVICEACCOUNT)
○ Assigned to the groups system
●
serviceaccounts
●
system:serviceaccounts:(NAMESPACE)
Service Account Tokens
Kubectl get serviceaccounts gitlab -o yaml

apiVersion: v1
kind: ServiceAccount
metadata:
# ...
secrets:
- name: gitlab-token-2wsdd

kubectl get secret gitlab-token-2wsdd -o yaml

apiVersion: v1
data:
ca.crt: (APISERVER'S CA BASE64 ENCODED)
namespace: ZGVmYXVsdA==
token: (BEARER TOKEN BASE64 ENCODED)
 Authentication
and Autherization:

Authorization with RBAC

Autherization Mode

● Node
○ A special-purpose authorization mode
○ Grants permissions to kubelets
○ API server option: --authorization-mode=Node
● Attribute-Based Access Control
○ Static file with a single “admin” identity
○ A simple file-based autherization policy
○ API server option: --authorization-mode=ABAC

● Role Based Access Control (RBAC)

○ API server option: --authorization-mode=RBAC

● Webhook
○ API server option: --authorization-mode=Webhook
API Server GET Requests
API Server POST Requests
API Server PATCH Requests
API Server DELETE Requests
API Workshop
Access to API
$ kubectl proxy
Starting to serve on 127.0.0.1:8001
● Access to API without $ curl localhost:8001/

authentication {
"paths": [
"/api",
"/api/v1",
...

$ curl localhost:8001/api/v1
...
{
"name": "services",
"singularName": "",
"namespaced": true,
"kind": "Service",
"verbs": [
"create",
...
API Structure

● API url to access with “curl”

/<api_basic_path>/<version>/<resources_all>
/<api_basic_path>/<version>/namespaces/<ns>/<rs_type>/<rs_name>
/<api_basic_path>/<version>/namespaces/<ns>/<rs_type>/<rs_name>/<subresource>

/api/v1/namespaces
/api/v1/namespaces/default/secrets/secret-example
/api/v1/namespaces/default/pods/pod-example/log
Create Pod with curl

$ curl -XPOST -H 'Accept: application/json' \

● Create first-pod by -H 'Content-Type: application/json' \
-d { "apiVersion": "v1", "kind": "Pod", \

directly request to "metadata": { "name": "first-pod", "labels": \

{ "role": "front", "app": "nginx" } }, \
"spec": { "containers": [ { "name": "web", \
api server "image": "nginx", "ports": [ { \
"containerPort": 80 } ] } ] } } \
localhost:8001/api/v1/namespaces/default/pods/
 Role Base Access Control:

Role and Cluster Role

User and Groups

● An authentication plugin returns the username and group(s)

● User doesn’t store any information anywhere
○ Actual user
○ Pods
● K8s used it to verfiy authorization
● Groups: Both human users and ServiceAccounts can belong to one
or more groups
● Builtin groups
○ system:unauthenticated
○ system:authenticated
○ system:serviceaccounts
○ system:serviceaccounts:<namespace>
RBAC: Role

● An RBAC Role contains rules that apiVersion: rbac.authorization.k8s.io/v1

represent a set of permissions kind: Role
metadata:
● sets permissions within a particular namespace: default
namespace name: pod-reader
● you have to specify the namespace rules:
- apiGroups: [""] # "" indicates the
core API group
resources: ["pods"]
verbs: ["get", "watch", "list"]
RBAC: ClusterRole
● An RBAC ClusterRole contains rules
that represent a set of permissions
● Is a non-namespaced resource
● Cluster role usage
○ define permissions on namespaced
resources and be granted within
individual namespace(s)
○ define permissions on namespaced
resources and be granted across all
namespaces
○ define permissions on cluster-scoped
resourcesok
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
# "namespace" omitted since
ClusterRoles are not namespaced
name: secret-reader
rules:
- apiGroups: [""] # "" indicates the
core API group
resources: ["secrets"]
verbs: ["get", "watch", "list"]
RBAC: RoleBinding

● Grants the permissions defined in a apiVersion: rbac.authorization.k8s.io/v1

role to a user or set of users kind: RoleBinding
metadata:
● Grants permissions within a specific name: read-pods
namespace namespace: default
● It holds a list of subjects (users, subjects:
- kind: User
groups, or service accounts) name: jane # "name" is case sensitive
● Usage apiGroup: rbac.authorization.k8s.io
roleRef:
○ Reference any Role in the same kind: Role #this must be Role or
namespace ClusterRole
○ reference a ClusterRole and bind name: pod-reader
apiGroup: rbac.authorization.k8s.io
that ClusterRole to the namespace
of the RoleBinding
RBAC: ClusterRoleBinding

● Grant permissions across a whole apiVersion: rbac.authorization.k8s.io/v1

cluster kind: ClusterRoleBinding
metadata:
● After you create a binding, you cannot name: read-secrets-global
change the Role or ClusterRole that it subjects:
- kind: Group
refers to
name: manager
● It holds a list of subjects (users, apiGroup: rbac.authorization.k8s.io
groups, or service accounts) roleRef:
kind: ClusterRole
name: secret-reader
apiGroup: rbac.authorization.k8s.io
 Authentication
and Autherization:

RBAC Workshop
Namespace Admin
apiVersion: rbac.authorization.k8s.io/v1
kind: Role $ kubectl create serviceaccount behrad
metadata: serviceaccount/behrad created
namespace: default
name: namespace-admin $ kubectl apply -f namespace-admin-role.yaml
rules: role.rbac.authorization.k8s.io/namespace-admin created
- apiGroups:
- "*"
verbs: $ kubectl apply -f namespace-admin-rolebinding.yaml
- "*" rolebinding.rbac.authorization.k8s.io/namespace-admin-behrad created
resources:
- "*"
$ kubectl config set-credentials behrad --token $TOKEN
apiVersion: rbac.authorization.k8s.io/v1 User "behrad" set.
kind: RoleBinding
metadata: $ kubectl config set-context default-admin --cluster kubernetes --user behrad
name: namespace-admin-behrad Context "default-admin" created.
namespace: default
roleRef: $ kubectl --context default-admin -n kube-system get pods
apiGroup: rbac.authorization.k8s.io Error from server (Forbidden): pods is forbidden: User
kind: Role
name: namespace-admin
"system:serviceaccount:default:behrad" cannot list resource
subjects: "pods" in API group "" in the namespace "kube-system"
- kind: ServiceAccount
name: behrad
namespace: default
Securing Cluster Nodes
Allowing the Pod to
Use the Host’s Linux
Namespaces
hostNetwork
 hostNetwork
apiVersion: v1 $ Kubectl exec -ti pod-with-hostnetwork ip address show
kind: Pod docker0
metadata: Link encap:Ethernet HWaddr 02:42:14:08:23:47
name: pod-with-hostnetwork inet addr:172.17.0.1 Bcast:0.0.0.0 Mask:255.255.0.0
spec: ...
lo
hostNetwork: true
...
containers: ens160
- name: main link/ether 00:50:56:9b:d4:c8 brd ff:ff:ff:ff:ff:ff
image: busybox inet 172.19.6.50/24 brd 172.19.6.255 scope .. ens160
command: ["/bin/sleep", "999999"]
hostPort
 hostPort
apiVersion: v1 $ Kubectl exec -ti pod-with-hostport ip address show
kind: Pod 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state
metadata: UNKNOWN qlen 1000
name: pod-with-hostport link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
spec:
valid_lft forever preferred_lft forever
containers: 2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN qlen
- image: nginx:stable-alpine 1000
name: nginx link/ipip 0.0.0.0 brd 0.0.0.0
ports: 4: eth0@if17: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu
- containerPort: 80 1440 qdisc noqueue state UP
hostPort: 80 link/ether 12:19:08:d8:30:06 brd ff:ff:ff:ff:ff:ff
inet 172.16.39.10/32 scope global eth0
protocol: TCP
valid_lft forever preferred_lft forever
 hostPID and hostIPC
apiVersion: v1 $ Kubectl exec -ti pod-with-host-pid-and-ipc ps aux
kind: Pod PID TTY STAT TIME COMMAND
metadata: 1 ? Ss 9:29 /sbin/init
name: pod-with-host-pid-and-ipc 2 ? S 0:01 [kthreadd]
4 ? I< 0:00 [kworker/0:0H]
spec:
6 ? I< 0:00 [mm_percpu_wq]
hostPID: true 7 ? S 3:09 [ksoftirqd/0]
hostIPC: true 8 ? I 18:47 [rcu_sched]
containers: 9 ? I 0:00 [rcu_bh]
- name: main ...
image: busybox
command: ["/bin/sleep", "999999"]
Security Context
Security Context

● Specify the user (the user’s ID)

● Prevent the container from running as root
● Run the container in privileged mode, giving it full access to the
node’s kernel
● Adding or dropping capabilities
● Prevent the process from writing to the container’s filesystem.
● Set SELinux
Security Context

● runAsNonRoot: true (Default: false)

● privileged: true (Default: false)
● capabilities:
○ add
○ drop
● readOnlyRootFilesystem: true (Default: false)
○ only allow them to write to mounted volumes
● fsGroup and supplementalGroups
 Security Context - privileged
apiVersion: v1 $ kubectl exec -ti pod-privileged ls /dev
kind: Pod autofs sda2 tty7
metadata: block sda3 tty8
name: pod-privileged bsg sda4 tty9
spec: btrfs-control sda5 ttyprintk
containers: bus serial ttyS0
- name: main char sg0 ttyS1
image: busybox console shm ttyS10
command: ["/bin/sleep", "999999"] core snapshot ttyS11
securityContext: cpu snd ttyS12eel)
privileged: true
 Security Context - runAsUser
apiVersion: v1 $ kubectl run test-box –image=buxybox
kind: Pod pod/test-box created
metadata: $ kubectl exec -ti test-box -- id
name: pod-as-user uid=0(root) gid=0(root) groups=10(wheel)
spec:
containers: $ kubectl apply -f pod-as-user.yaml
- name: main pod/pod-as-user created
image: busybox $ kubectl exec -ti pod-as-user -- id
command: ["/bin/sleep", "999999"] uid=1001 gid=0(root)
securityContext:
runAsUser: 1001
 Security Context - capabilities
apiVersion: v1 $ kubectl apply -f pod-settime.yaml
kind: Pod pod/test-box created
metadata: $ kubectl exec -ti pod-settime -- date -s "12:00:00"
name: pod-settime 12:00:00
spec: $ kubectl exec -ti pod-settime – date
containers: Sun Aug 30 12:00:05 +0430 2020
- name: main
image: busy-box
command: ["/bin/sleep", "999999"]
securityContext:
capabilities:
add:
- SYS_TIME
Docker Default Capabilities

● These capabilities can be drop

○ SETPCAP Modify process capabilities.
○ MKNOD Create special files using mknod(2).
○ AUDIT_WRITE Write records to kernel auditing log.
○ CHOWN Make arbitrary changes to file UIDs and GIDs (see chown(2)).
○ NET_RAW Use RAW and PACKET sockets.
○ DAC_OVERRIDE Bypass file read, write, and execute permission checks.
○ FOWNER Bypass permission checks on operations that normally require the file
system UID of the process to match the UID of the file.
○ FSETID Don’t clear set-user-ID and set-group-ID permission bits when a file is
modified.
○ KILL Bypass permission checks for sending signals.
○ ...
Docker Default Capabilities

● These capabilities can be drop

○ KILL Bypass permission checks for sending signals.
○ SETGID Make arbitrary manipulations of process GIDs and supplementary GID
list.
○ SETUID Make arbitrary manipulations of process UIDs.
○ NET_BIND_SERVICE Bind a socket to internet domain privileged ports (port
numbers less than 1024).
○ SYS_CHROOT Use chroot(2), change root directory.
○ SETFCAP Set file capabilities.
 Security Context – Share Volume
apiVersion: v1 $ kubectl apply -f pod-with-shared-volume.yaml
kind: Pod pod/test-box created
metadata: $ kubectl exec -ti pod-with-shared-volume -c first
name: pod-with-shared-volume sh
spec: /$ id
securityContext: uid=1111 gid=0(root) groups=555,666,777
fsGroup: 555
supplementalGroups: [666, 777]
containers:
- name: first
image: busybox
command: ["/bin/sleep", "999999"]
securityContext:
runAsUser: 1111
volumeMounts:
- name: shared-volume
mountPath: /volume
...
volumes:
- name: shared-volume
emptyDir:
Pod Security Policy
PodSecurityPolicy (psp)

● Restricting the use of security-related features in pods

● Is a cluster-level (non-namespaced) resource
● The PodSecurityPolicy is a admission control plugin
● A PodSecurityPolicy resource defines:
○ Whether a pod can use the host’s IPC, PID, or Network namespaces
○ Which host ports a pod can bind to
○ What user IDs a container can run as
○ Whether a pod with privileged containers can be created
○ Which kernel capabilities are allowed, default, always dropped
○ What SELinux labels a container can use
○ Whether a container can use a writable root filesystem or not
○ Which filesystem groups the container can run as
○ Which volume types a pod can use
Enabling Pod Security Policy
apiVersion: kubeadm.k8s.io/v1beta2
● Enable when init cluster kind: ClusterConfiguration
kubernetesVersion: v1.18.6
with kubeadm apiServer:
● Enable by edit kube- extraArgs:
advertise-address: 192.168.43.202
apiserver manifests enable-admission-plugins:
NodeRestriction,PodSecurityPolicy
...

$ vi /etc/kubernetes/manifests/kube-apiserver.yaml
...
- --enable-admission-
plugins=NodeRestriction,PodSecurityPolicy
...
 Default Pod Security Policy
apiVersion: extensions/v1beta1 $ kubectl create clusterrole psp-default \
kind: PodSecurityPolicy --verb=use --resource=podsecuritypolicies \
metadata: --resource-name=default
name: default clusterrole "psp-default" created
spec:
hostIPC: false $ kubectl create clusterrolebinding psp-all-users \
hostPID: false --clusterrole=psp-default \
hostNetwork: false --group=system:authenticated
hostPorts: clusterrolebinding.rbac.authorization.k8s.io/psp-
- min: 10000 all-users created
max: 11000
- min: 13000
max: 14000
privileged: false
readOnlyRootFilesystem: true
runAsUser:
rule: RunAsAny
fsGroup:
rule: RunAsAny
…
runAsUser, fsGroup, and supplementalGroups
runAsUser: $ kubectl create -f pod-as-user-100.yaml
rule: MustRunAs Error from server (Forbidden): error when creating
ranges: "pod-as-user-100.yaml"
- min: 2 : pods "pod-as-user-guest" is forbidden: unable to
max: 2 validate against any pod
fsGroup: security policy: [securityContext.runAsUser: Invalid
rule: MustRunAs value: 100: UID on
ranges: container main does not match required range. Found
- min: 2 100, allowed: [{2 2}]]
max: 10
- min: 20
max: 30
supplementalGroups:
rule: MustRunAs
ranges:
- min: 2
max: 10
- min: 20
Max: 30
 Capabilities
apiVersion: extensions/v1beta1 $ kubectl create -f dep-ingress-nginx-hostnet.yaml
kind: PodSecurityPolicy Error from server (Forbidden): error when creating
spec: "dep-ingress-nginx-hostnet.yaml": pods "ingress-
allowedCapabilities: nginx-controller" is forbidden: unable
- SYS_TIME to validate against any pod security policy:
defaultAddCapabilities: [capabilities.add: Invalid
- CHOWN value: "NET_ADMIN": capability may not be added]
requiredDropCapabilities:
- SYS_ADMIN
- SYS_MODULE
- NET_ADMIN
 HostPaths
apiVersion: extensions/v1beta1 $
kind: PodSecurityPolicy
spec:
allowedHostPaths:
# This allows "/foo", "/foo/",
"/foo/bar" etc., but
# disallows "/fool", "/etc/foo" etc.
# "/foo/../" is never valid.
- pathPrefix: "/foo"
readOnly: true # only allow read-only
mounts