Certified Kubernetes Administrator

(CKA)
Behrad Eslamifar
b.eslamifar@gmail.com
Logging/Monitoring
5%
Outlines

● Understand how to monitor all cluster components.

● Understand how to monitor applications.
● Manage cluster component logs.
● Manage application logs.
 Kubernetes
Monitoring Atchitecture
Kubernetes Monitoring Architecture

● A core metrics pipeline consisting of Kubelet

○ These metrics are used by core system components
○ This pipeline is not intended for integration with third-party monitoring systems
● A monitoring pipeline used for collecting various metrics from the
system and exposing them to end-users,
○ Kubernetes will not ship with a monitoring pipeline
○ expect that such pipelines will typically consist of a per-node agent and a cluster-
level aggregator
Type of Metrics

● System Metrics
○ generic metrics that are generally available from every entity that is monitored
(e.g. usage of CPU and memory by container and node
● Service Metrics
○ are explicitly defined in application code and exported (e.g. number of 500s
served by the API server)
System Metrics

● Core Metrics
○ Which are metrics that Kubernetes understands and uses for operation of its
internal components and core utilities
● Non-Core Metrics
○ Which are not interpreted by Kubernetes
○ Assume they include the core metrics (though not necessarily in a format
Kubernetes understands)
 Evolution of components to monitor
https://www.datadoghq.com/blog/monitoring-kubernetes-era
Components that Now Need to be Monitored

● Your hosts, even if you don’t know which containers and

applications they are actually running
● Your containers, even if you don’t know where they’re running
● Your containerized applications
● The Kubernetes cluster itself
Exporters

● metrics-server – CPU, memory, file-descriptors, disks, etc of the

cluster
● cAdvisor – a Docker daemon metrics – containers monitoring
● kube-state-metrics – deployments, pods, nodes
● node-exporter: EC2 instances metrics – CPU, memory, network
 Kubernetes
Monitoring Atchitecture

Monitoring Workshop
 Kubernetes Metric Server

$ cd Kubernetes-Certified-Administrator
● Collects resource metrics from $ git pull
$ cd labs/monitoring/manifests/
Kubelets $ kubectl apply -f metric-server.yaml
serviceaccount/metrics-server created
● Exposes them in Kubernetes clusterrole.rbac.authorization.k8s.io/system:aggregated-
metrics-reader created
clusterrole.rbac.authorization.k8s.io/system:metrics-server
apiserver through Metrics API created
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-
● Use by Horizontal Pod reader created
...
Autoscaler and Vertical Pod $ kubectl top pods

Autoscaler NAME
mariadb-0
CPU(cores)
2m
MEMORY(bytes)
74Mi
web-server-6d48d5449c-75rj6 0m 3Mi
web-server-6d48d5449c-fg95l 0m 3Mi
wordpress-756bb74786-wghf7 0m 0Mi

$ kubectl top node

NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
master-1 340m 17% 1261Mi 66%

* https://github.com/kubernetes-sigs/metrics-server/releases
Metric Server Use Cases: Autoscaling

● CPU/Memory based apiVersion: autoscaling/v1

kind: HorizontalPodAutoscaler
horizontal autoscaling metadata:
name: autoscale-for-deployment
● Automatically spec:
maxReplicas: 5
adjusting/suggesting resources minReplicas: 1
scaleTargetRef:
needed by containers apiVersion: apps/v1
kind: Deployment
(Vertical Pod Autoscaler) name: test-deployment
TargetCPUUtilizationPercentage: 30
 kube-state-metrics
$ kubectl apply -f service-account.yaml
$ kubectl apply -f cluster-role.yaml
● Collects resource metrics from $ kubectl apply -f cluster-role-binding.yaml
$ kubectl apply -f service.yaml
Kubelets $ kubectl apply -f deployment.yaml

● Exposes them in Kubernetes

apiserver through Metrics API
● Use by Horizontal Pod
Autoscaler and Vertical Pod
$ kubectl exec -ti test-box -- sh
/ # curl http://kube-state-metrics.kube-
system.svc.cluster.local:8080/metrics
kube_configmap_info{namespace="kube-system",configmap="calico-
config"} 1
kube_configmap_info{namespace="kube-
system",configmap="coredns"} 1

Autoscaler ...

* https://github.com/kubernetes/kube-state-metrics
/ # curl http://kube-state-metrics.kube-
system.svc.cluster.local:8081/metrics
# HELP process_resident_memory_bytes Resident memory size in
bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 3.34848e+07
# HELP process_start_time_seconds Start time of the process
since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.60637852531e+09
Node Exporter
$ kubectl apply -f kubectl apply -f node-exporter-
daemonset.yaml
● Prometheus exporter for namespace/monitoring unchanged
clusterrole.rbac.authorization.k8s.io/node-exporter created
hardware and OS metrics clusterrolebinding.rbac.authorization.k8s.io/node-exporter
created
exposed by *NIX kernels service/node-exporter created
serviceaccount/node-exporter created
daemonset.apps/prometheus-node-exporter created

$ kubectl exec -ti test-box -- sh

/ # curl
http://node-exporter.monitoring.svc.cluster.local:9100/metrics
...
HELP node_cpu_seconds_total Seconds the cpus spent in each
mode.
# TYPE node_cpu_seconds_total counter
node_cpu_seconds_total{cpu="0",mode="idle"} 58111
node_cpu_seconds_total{cpu="0",mode="iowait"} 36.24
node_cpu_seconds_total{cpu="0",mode="irq"} 0
node_cpu_seconds_total{cpu="0",mode="nice"} 0.02
node_cpu_seconds_total{cpu="0",mode="softirq"} 313.64
...

* https://github.com/prometheus/node_exporter
 Prometheus
● An open-source monitoring
system with a dimensional
data model, flexible query
$ kubectl apply -f prometheus-deployment.yaml
serviceaccount/prometheus created
service/prometheus-service created
clusterrole.rbac.authorization.k8s.io/prometheus created
clusterrolebinding.rbac.authorization.k8s.io/prometheus created
configmap/prometheus-config created
deployment.apps/prometheus-deployment created

language, efficient time

*** browse http://node-ip:30900 ***
series database and modern
alerting approach.

* https://github.com/prometheus/prometheus
* https://prometheus.io/
 Kubernetes Dashboard
● Kubernetes Dashboard is a
general purpose, web-based
UI for Kubernetes clusters
* https://github.com/kubernetes/dashboard
$ kubectl apply -f dashboad.yaml
namespace/kubernetes-dashboard created
serviceaccount/kubernetes-dashboard created
service/kubernetes-dashboard created
secret/kubernetes-dashboard-certs created
...
$ kubectl apply -f dashboad-admin.yaml
serviceaccount/dashboard-admin created
clusterrolebinding.rbac.authorization.k8s.io/dashboard-admin
created
*** browse http://node-ip:30910 ***
 Kubernetes
Logging Architecture
Kubernetes Logging Architecture

● Node level logging

○ Docker logging driver
○ Logrotate consideration
○ K8s systemd based components (e.g. kubelet, ...)
● Cluster level logging
○ If a container crashes, a pod is evicted, or a node dies, you'll usually still want to
access your application's logs (Concept)
○ Kubernetes does not provide a native solution for cluster-level logging
○ Logrotate consideration done with solution
 Node Level Logging
# cat /etc/docker/daemon.json
{
"log-driver": "json-file",
"log-opts": {
"max-size": "10m",
"max-file": "5"
}
}
Cluster Level: Node Logging Agent
Cluster Level: Streaming Sidecar Container
Cluster Level: Sidecar with Logging Agent
Exposing logs directly from the application
Kubernetes Components

● Master
○ API Server
○ Scheduler
○ Controller manager
● Node
○ kubelet (/var/log/kubelet.log , journalctl -u kubelet)
○ kube-proxy
Auditlog

● what happened?
● when did it happen?
● who initiated it?
● on what did it happen?
● where was it observed?
● from where was it initiated?
● to where was it going?
Auditlog: Policy Level

● None - don't log events that match this # Log all requests at the Metadata level.
rule. apiVersion: audit.k8s.io/v1
kind: Policy
● Metadata - log request metadata rules:
- level: Metadata
(requesting user, timestamp, resource,
verb, etc.) but not request or response
body.
● Request - log event metadata and
request body but not response body.
This does not apply for non-resource
requests.
● RequestResponse - log event
metadata, request and response
bodies. This does not apply for non-
resource requests.
Auditlog: Policy Stage
apiVersion: audit.k8s.io/v1 # This is required.
kind: Policy
● RequestReceived - The stage for # Don't generate audit events for all requests in
events generated as soon as the audit RequestReceived stage.
omitStages:
handler receives the request, and - "RequestReceived"
rules:
before it is delegated down the handler # Log pod changes at RequestResponse level
- level: RequestResponse
chain. resources:
● ResponseStarted - Once the response - group: ""
# Resource "pods" doesn't match requests to
headers are sent, but before the any subresource of pods,
# which is consistent with the RBAC policy.
response body is sent. This stage is resources: ["pods"]
...
only generated for long-running # A catch-all rule to log all other requests at
the Metadata level.
requests (e.g. watch). - level: Metadata
● ResponseComplete - The response # Long-running requests like watches that
fall under this rule will not
body has been completed and no more # generate an audit event in RequestReceived.
omitStages:
bytes will be sent. - "RequestReceived"

● Panic - Events generated when a panic

occurred.
Auditlog: Log Backend
- ... --audit-policy-file=/etc/kubernetes/audit-
policy.yaml
● The log backend writes audit - --audit-log-path=/var/log/apiserver/audit.log
- --audit-log-maxage=30

events to a file in JSONlines - --audit-log-maxsize=200

- --audit-log-maxbackup=10
...
format. volumeMounts:
- mountPath: /etc/kubernetes/audit-policy.yaml
name: audit
● configure the log audit readOnly: true
- mountPath: /var/log/apiserver

backend using the kube- name: audit-log

readOnly: false
...
apiserver flags - name: audit
hostPath:
path: /etc/kubernetes/audit-policy.yaml
type: File

- name: audit-log
hostPath:
path: /var/log/apiserver
type: DirectoryOrCreate