Getting Started With GKE

RENU SOOD, Author, IBM Corporation

SIDDHARTHA SOOD, Reviewer, IBM Corporation

Abstract: This paper describes step by step approach on how to deploy Jenkins and use Jenkins
pipeline to deploy a simple hello world microservice on GKE cluster on Google Cloud.
Category: Modernization
General Terms: GKE
Additional Keywords and Phrases: Microservices, DevOps , Modernization, Jenkins
Technology used: GCP (Google Cloud Platform), GCloud SDK, Helm, Jenkins, Kubernetes plugin

INTRODUCTION
Google Kubernetes Engine provides a managed environment for deploying, managing, and scaling
enterprise containerized applications using Google infrastructure. GKE environment consists of
multiple machines (specifically, Google Compute Engine instances) grouped together to form a
cluster.

GKE clusters are powered by the Kubernetes open source cluster management system. Kubernetes
provides the mechanisms through which we interact with our GKE cluster. We use Kubernetes
commands and resources to deploy and manage our enterprise containerized applications, perform
administration tasks, set relevant policies, and monitor the health of our deployed workloads.

Kubernetes on Google Cloud Platform:

With a GKE cluster, we also gain the benefit of advanced cluster management features that Google
Cloud Platform provides. These include:
 Google Cloud Platform's load-balancing for Compute Engine instances
 Node pools to designate subsets of nodes within a cluster for additional flexibility
 Automatic scaling of our GKE cluster's node instance count
 Automatic upgrades for our GKE cluster's node software
 Node auto-repair to maintain node health and availability
 Logging and Monitoring with Stackdriver for visibility into our GKE cluster

Getting started with GKE Page 1 of 44

Working with GCP
 Setup free Tier Account ( Note: A free tier account was used for this paper)
 Use GCP Console: https://console.cloud.google.com
 For command line interface (cli) use Google cloud shell (This can be leveraged from google
cloud console as shown below or by installing google cloud SDK).

Using Gcloud:
To work with GCP resources using command line you have option to use cloud shell or google cloud
SDK. Which can be downloaded from following URL:
https://cloud.google.com/appengine/docs/standard/go/download

Using Cloud SDK

After installing Google cloud SDK, you can run cloud shell from you desktop. Run Gcloud init , to
initialize your work environment.

Using Cloud SDK : setting up KUBECTL

Kubectl is command line utility and is used to execute commands and manage Kubernetes cluster
After initialing gcloud shell, we would install kubectl using following command: gcloud
components install kubectl

Getting started with GKE Page 2 of 44

USE CASE:
Deploy Jenkins on Kubernetes and use Jenkins pipeline to deploy simple Microservices on
Kubernetes.

BASIC CONCEPTS OF DEVOPS

DevOps: The term DevOps is derived from the combination of two words: developer and
operations. The goal of DevOps is to reduce the time needed from the identification of a new
requirement to the time it goes live for customers. The DevOps journey introduces such practices as
continuous delivery, which helps to reduce time to produce better quality software.
Continuous integration (CI): It is development practice that requires the developer to integrate
code in a central shared repository. Every time developer commits the code, it is integrated with
other code and verified by execution of the test. CI starts every time we commit code to the central
shared repository. Adopting CI is cheap, essentially, we only need a server with Jenkins, and we can
start to use it. CI can be summarized in three phases: Develop, Test, Deploy
Continuous Development: CD is software engineering practice used to release software within a
short cycle.
Continuous Deployment: Continuous deployment and continuous delivery are similar, but there
is a substantial difference between the two. By Continuous delivery we are referring to a pipeline
for creating build but not necessarily one we intend to release to production. With continuous
deployment, we release the build to production every time.
Differences Between Continuous Integration and Continuous Delivery:
CI and CD are similar, but there are some differences between these two practices.

 CI concentrates on integrating the software with every commit.

 CD extends CI because it adds another layer after integration and testing of software are
complete.

 CD build the software and prepares it for potential release

Development Pipeline: To ensure a successful DevOps journey, one of the most important jobs is
to define the development pipeline. Building this pipeline is essentially the core of the changes
required by DevOps
Containerization with Docker and Kubernetes: Containerization is an evaluation of
virtualization. With virtualization we usually re-create an entire operating system (OS) and host it
on a host machine. Using a software for the container such as Docker, it is possible to create a
complete image of our application and release it via the common registry. To manage and release
this image, we can use software know as container orchestrator, such as Kubernetes.
Why use Docker: Docker becomes more popular with the cloud, because it is very useful when we
have to build containerized applications. There are other containerization technologies as well but
docker is the most popular.

Getting started with GKE Page 3 of 44

What is Kubernetes Engine: Kubernetes sometimes referred as K8S is an open source platform
developed by google and managed by the cloud Native Computing foundation. It is used to manage
and scale containerized applications in a cluster, such as docker.

Important concepts related to Kubernetes and GKE

Source - https://cloud.google.com/blog/products/gcp/google-kubernetes-engine-1-10-is-generally-
available-and-ready-for-the-enterprise

Cluster
A cluster is the foundation of GKE: the Kubernetes objects that represent containerized applications
all run on top of a cluster. In Google Kubernetes Engine, a cluster consists of at least one cluster
master and multiple worker machines called nodes. These master and node machines run the
Kubernetes cluster orchestration system.
Cluster master
The cluster master runs the Kubernetes control plane processes, including the Kubernetes API
server, scheduler, and core resource controllers. The master's lifecycle is managed by GKE when we
create or delete a cluster. This includes upgrades to the Kubernetes version running on the cluster

Getting started with GKE Page 4 of 44

master, which GKE performs automatically, or manually as requested in situations when we prefer
to upgrade earlier than the automatic schedule.
Cluster master and the Kubernetes API
The master is the unified endpoint for the cluster. All interactions with the cluster are done via
Kubernetes API calls, and the master runs the Kubernetes API Server process to handle those
requests. We can make Kubernetes API calls directly via HTTP/gRPC, or indirectly, by running
commands from the Kubernetes command-line client (kubectl) or interacting with the UI in the GCP
Console.
The cluster master's API server process is the hub for all communication for the cluster. All internal
cluster processes (such as the cluster nodes, system and components, application controllers) all
act as clients of the API server; the API server is the single "source of truth" for the entire cluster.
Master and node interaction
The cluster master is responsible for deciding what runs on all the cluster's nodes. This can include
scheduling workloads, like containerized applications, and managing the workloads' lifecycle,
scaling, and upgrades. The master also manages network and storage resources for those
workloads. The master and nodes also communicate using Kubernetes APIs. Master interactions
with the gcr.io container registry.
When we create or update a cluster, container images for the Kubernetes software running on the
masters (and nodes) are pulled from the gcr.io container registry. An outage affecting the gcr.io
registry may cause the following types of failures:

 Creating new clusters will fail during the outage.

 Upgrading clusters will fail during the outage.

Nodes: A cluster typically has one or more nodes, which are the worker machines that run our
containerized applications and other workloads. The individual machines are Compute Engine VM
instances that GKE creates on our behalf when we create a cluster.
Each node is managed from the master, which receives updates on each node's self-reported status.
A node runs the services necessary to support the Docker containers that make up your cluster's
workloads. These include the Docker runtime and the Kubernetes node agent (kubelet) which
communicates with the master and is responsible for starting and running Docker containers
scheduled on that node.
In GKE, there are also a number of special containers that run as per-node agents to provide
functionality such as log collection and intra-cluster network connectivity.

Getting started with GKE Page 5 of 44

Getting started with GKE Page 6 of 44
Namespaces: Namespaces is a logical division of our deployment in the cluster. It is important to
have different resources logically organized in the cluster. Using namespaces user can :uniquely
name resources ,delegated managed authority to defined users, limit the resource consumptions by
defining quotas for namespaces.
To check existing namespaces in Kubernetes: use the below commands

kubectl get namespace

kubectl describe namespace default

Creating namespace using yaml file.

Steps:

 Create a file called namespace.yaml .

 Copy below lines and save it.
 Execute the following commands : kubectl apply –file namespace.yaml

Contents of namespace.yaml:
kind: Namespace
apiVersion: v1
metadata:
name: devopsgcpnamespace
labels:
name: devopsgcpnamespace

GKE Workloads: GKE works with containerized applications: applications packaged into hardware
independent, isolated user-space instances, for example by using Docker. In GKE and Kubernetes,
these containers, whether for applications or batch jobs, are collectively called workloads. Before
we deploy a workload on a GKE cluster, we must first package the workload into a container.
Pods: A pod is smallest deployment unit in the kubernetes world. It is a group of one or more
containers for eg, docker, with shared network/storage. To verify the pods deployed on Kubernetes
: Kubectl get pods

To check the status of the pod : Kubectl describe <podname>

Deployments: Deployments are used to declare and manage the state of the pods and ReplicaSet.
Deployments is responsible for creating and updating instances of our containerized application.

Getting started with GKE Page 7 of 44

Once the Deployment has been created, the Kubernetes master schedules the application instances
onto individual nodes in the cluster. Deployment is high level of abstraction ,it manages the
ReplicaSets when doing Pod Orchestration, creation, deletion and updates. A deployment provides
declarative updates for Pods and ReplicaSets. The Deployment allows for easy updating of a
ReplicaSet as well as the ability to roll back to previous deployment. The main purpose of
deployments is to do rolling updates and rollbacks.
Sample manifest file for deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
namespace: devopsgcpnamespace
labels:
app: nginx
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:2.7.9
ports:
- containerPort: 80

Both Pod and Deployment are full-fledged objects in the Kubernetes API. Deployment manages
creating Pods by means of ReplicaSets. What it boils down to is that Deployment will create Pods
with spec taken from the template. It is rather unlikely that you will ever need to create Pods
directly for a production use-case.
Services: A service is an abstraction that defines logical group of pods and a policy for accessing
them. .
Exposing a service: kubectl expose deployment test --target-port=80 -type=NodePort
To verify the services: kubectl get svc --namespace jenkinsgcp
Services Types determines how the service will available outside cluster, service type can be on of
the following :

 ClusterIP: service is reachable within Kubernetes Cluster (services that are not to be
exposed to external world should be configured as ClusterIP.
 NodePort: Service is reachable through NodeIP:NodeportAddress
 LoadBalancer: Service is reachable though an external load balancer mapped to
<NODEIP>:NodePort Adress

Getting started with GKE Page 8 of 44

Sample manifest file for exposing the service
kind: Service
apiVersion: v1
metadata:
name: demo
spec:
type: LoadBalancer
ports:
- name: http
port: 8080
targetPort: 8080
protocol: TCP
selector:
app: demo
role: api

Quotas: Quotas are a way to limit the usage of resources across the kubernetes cluster. A quota is
defined using object ResourceQuata : CPU, MEMORY<Storage>

Getting started with GKE Page 9 of 44

Jenkins
Jenkins is an open source automation tool written in Java with plugins built for Continuous
Integration purpose. Jenkins is used to build and test your software projects continuously making it
easier for developers to integrate changes to the project and making it easier for users to obtain a
fresh build. It also allows you to continuously deliver your software by integrating with many
testing and deployment technologies. With Jenkins, organizations can accelerate the software
development process through automation. Jenkins integrates development life-cycle processes of
all kinds, including build, document, test, package, stage, deploy, static analysis and much more.
Jenkins achieves Continuous Integration with the help of plugins. Plugins allows the integration of
Various DevOps stages. If you want to integrate a particular tool, you need to install the plugins for
that tool. For example: Git, Maven, Kubernetes etc.
It is an open source tool with great community support. It is easy to install. It has 1000+ plugins to
ease your work. If a plugin does not exist, you can code it and share with the community. It is free of
cost. It is built with Java and hence, it is portable to all the major platforms.

Getting started with GKE Page 10 of 44

IMPLEMENTATION:
Use Case, deploy Jenkins on Kubernetes deploy simple Microservices API on Kubernetes using
Jenkins pipeline, as per the diagram below:

Getting started with GKE Page 11 of 44

Steps to automate and deploy

 Write and check-in a deployment unit in code repository (a small microservice)

 Setup Kubernetes cluster
 Setup Jenkins
 Create a pipeline
 Setup build environment in Jenkins
 Execute the pipeline

Code for developing microservice:

Develop a sample microservice with /helloworld or /sayGoodMorning and check in the code in
GCP source repository and proceed with below steps

Using google cloud source repository

Google provides private GIT repository that can be used for source code management
(https://cloud.google.com/source-repositories/)

To start using google cloud repository you need to enable the source repository, follow the below
steps to enable and start using google cloud source repository:

 login to gcp
 From the projects list, select a project or create a new one
 Go to api library page and enable google cloud source repository

Getting started with GKE Page 12 of 44

Once api is enabled , use the following link to navigate to google cloud repository
https://source.cloud.google.com/repos , and create your repository and check-in the code.

Commands to push code to google repository

 gcloud init
 gcloud source repos clone TestApp --project=mystical-runway-247816
 cd TestApp
 git add .
 git –m commit “base version”
 git push -u origin master

Getting started with GKE Page 13 of 44

Setting up cluster on Google cloud
Execute the below command in cloud shell to create cluster:
gcloud container clusters create <jenkins-cd> --zone us-east1-b --scopes
https://www.googleapis.com/auth/projecthosting,storage-rw,https://www.googleapis.com/
auth/source.full_control,cloud-platform,compute-rw,storage-full
Once the above command is executes successfully, verify your cluster is setup using : kubectl
cluster-info
To connect to your cluster in gcloud , use the following command : gcloud container clusters get-
credentials jenkins-cd
To setup sample application on the above cluster: kubectl run hello-server --image gcr.io/google-
samples/hello-app:1.0 --port 8080
To expose the deployment as a service: kubectl expose deployment hello-server --type
"LoadBalancer"
To verify expose service : kubectl get service hello-server

Getting started with GKE Page 14 of 44

 Setting up Jenkins on Kubernetes

1)Create Kubernetes cluster (done in above steps )

Verify the setup cluster : gcloud container clusters list
2) Clone the sample code, or download the zip file : git clone
https://github.com/GoogleCloudPlatform/continuous-deployment-on-kubernetes.git
The Git repository contains Kubernetes manifests that you'll use to deploy Jenkins. The manifests
and their settings are described in Configuring Jenkins for GKE.
3) cd continuous-deployment-on-kubernetes
4) Installing Helm : Use Helm to deploy Jenkins from the repository
5) Download and install the Helm binary: wget
https://storage.googleapis.com/kubernetes-helm/helm-v2.14.1-linux-amd64.tar.gz
6) Unzip the file to your local system: tar zxfv helm-v2.14.1-linux-amd64.tar.gz
7) cp linux-amd64/helm .
8) Add yourself as a cluster administrator in the cluster's RBAC so that you can give Jenkins
permissions in the cluster: kubectl create clusterrolebinding cluster-admin-binding --
clusterrole=cluster-admin --user=$(gcloud config get-value account)
9) Grant Tiller, the server side of Helm, the cluster-amin role in your cluster: kubectl create
serviceaccount tiller --namespace kube-system kubectl create clusterrolebinding
tiller-admin-binding --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
10) Initialize Helm. This ensures that the Tiller is properly installed in your cluster.
11) Ensure Helm is properly installed by running the following command:
./helm version
12) Installing Jenkins
You will use a custom values file to configure the Jenkins installation. For details on the
configuration, look at the jenkins/values.yaml file.
Install Jenkins using the below command :
./helm install -n cd stable/jenkins -f jenkins/values.yaml --version 1.2.2 --wait

13) After that command completes, ensure the Jenkins pod goes to the Running state and the
container is in the READY state: kubectl get pods

Getting started with GKE Page 15 of 44

14) Set up port forwarding to the Jenkins UI from Cloud Shell:

export POD_NAME=$(kubectl get pods --namespace default \

-l "app.kubernetes.io/component=jenkins-master" -l "app.kubernetes.io/instance=cd“
-o jsonpath="{.items[0].metadata.name}") \
kubectl port-forward $POD_NAME 8080:8080 >> /dev/null &
15) Check that the Jenkins Service was created properly. Using : kubectl get svc
Connecting to Jenkins

16) Retrieve the admin password that was automatically created by the Jenkins Helm chart:

printf $(kubectl get secret cd-jenkins -o jsonpath="{.data.jenkins-admin-password}" | base64 --

decode);echo
17) To open the Jenkins user interface, click Web Preview in Cloud Shell and click Preview on port
8080.
18) Click log in on the top right of the window.
Enter admin for the User field and the password value from the previous step for the Password
field.
(reference url : https://cloud.google.com/solutions/jenkins-on-kubernetes-engine-tutorial)

Getting started with GKE Page 16 of 44

Deploying the application checked in the repository on Kubernetes
using Jenkins pipeline

The sample application exposes sayHello service which prints hello message and take employee
name from config maps(for config maps refer to page 31 onwards)
The Demo app checked in contains k8 folder which has two manifest files : …/k8s/dev/frontend-
dev.yaml : which contains deployment
…/k8s/services/ frontend.yaml , which exposes deployment as a service

Jenkins Pipeline Jenkinsfile

In Jenkins, a pipeline is a group of events or jobs which are interlinked with one another in a
sequence. In simple words, Jenkins Pipeline is a combination of plugins that support the integration
and implementation of continuous delivery pipelines using Jenkins
The Jenkins file explained in next few sections is enclosed below :

Jenkins Pipeline (Jenkinsfile) : Section 1 of Jenkins pipeline (defining templates)

Getting started with GKE Page 17 of 44

Jenkins Pipeline (Jenkinsfile) : Section 2 of Jenkins pipeline (volumes)

Getting started with GKE Page 18 of 44

Jenkins Pipeline (Jenkinsfile) : Section 3 of Jenkins pipeline (defining environment
variables)

Getting started with GKE Page 19 of 44

Jenkins Pipeline (Jenkinsfile) : Section 4 of Jenkins pipeline (git branch details and
container specific )

Getting started with GKE Page 20 of 44

Defining Dockerfile
Dockerfile is kind of a recipe to build an image. It’s a plain text file containing instructions
which are executed by docker in the order they are placed. Each dockerfile has a base image that
the Docker engine will use to build image. The image creation flow in docker consists of two steps.

You execute the docker build command to create a Docker image based on the Dockerfile

Getting started with GKE Page 21 of 44

Steps to configure Jenkins pipeline in jenkins
Generating credentials to connect to your gcp service account
 Login to Jenkins, select credentials from left menu , click on global from the screen that
appears (like below) , select add credentials from left menu

 select add credentials from left menu and you will get below screen , and in the kind drop
down select google Service Account from metadata

Getting started with GKE Page 22 of 44

  Click ok on screen similar to below and your gcp credentials will be created with name as
project name as show below

 Login to Jenkins

 Select New Item from top left corner, you will get screen like below screenshot, enter the
name of your pipeline and select multibranch pipeline , and click Ok.

Getting started with GKE Page 23 of 44

Enter the name of google source code repository example :
https://source.developers.google.com/p/test/r/demo
Select the credentials , and click on save

The Jenkins will connect to Google source repository, look for branches configured and scan the
Jenkins pipeline and build the code and then deploy. Giving access to Jenkins id to deploy and scale
the pods. Jenkins will deploy the artifacts on GCP using default Jenkins id (incase the deployment
fails with access issue, you can use define role based access to default user , by creating yaml file as
below and execute the same using
kubectl command : kubectl apply -f <your yaml file>

Getting started with GKE Page 24 of 44

Building Pipeline Select the pipeline you created and choose the branch you want to build. click on
build now (from left menu).The pipeline starts executing and if all goes ok, it will deploy the
changes you made to the pod.

Getting started with GKE Page 25 of 44

Output of buid pipeline

Getting started with GKE Page 26 of 44

The pods deployed to Kubernetes cluster

Getting started with GKE Page 27 of 44

The service exposed by the pod

Testing the service : To access the service use following URL :

http://35.196.222.178:8080/Test/sayHello/

Getting started with GKE Page 28 of 44

Setting up Ingress (load balancer for Jenkins)
Ingress is a Kubernetes resource that encapsulates a collection of rules and configuration for
routing external HTTP(S) traffic to internal services. On GKE, Ingress is implemented using Cloud
Load Balancing. When you create an Ingress in your cluster, GKE creates an HTTP(S) load balancer
and configures it to route traffic to your application. Ingress in it self is not a load balancer , or
physical deivice ,it’s a logical controller that will be mapped to one of the existing load balancer on
cloud and creates set of rules, and route traffic to appropriate endpoint.

The following manifest file defines an Ingress resource that directs traffic to your web Service

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: basic-ingress

annotations:
spec:
  backend:
    serviceName: web
    servicePort: 8080

Once you have created the configuration for load balance you verify using following command:
Kubectl get ingress
Kubectl describe ingress basic-ingress
Setting up Ingress with Static IP (load balancer)
Configuring a static IP address (optional)
Commands
To create static IP : gcloud compute addresses create jenkins-static-ip --global
To verify the IPs configured: gcloud compute addresses list
Once the static ip is configured you can specify the same in the yaml and update it
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: basic-ingress
  annotations:
    kubernetes.io/ingress.global-static-ip-name: " jenkins-static-ip"
spec:
  backend:
    serviceName: web
    servicePort: 8080

Serving multiple applications on a Load Balancer:

Getting started with GKE Page 29 of 44

Ingress, based on routing rules allows access multiple services deployed on GKE. In the current
example we have Jenkins deployed as one service and sample Demo app deployed as one service,
and need is to access both services from ingress load balancer, this can be done by writing different
rules in manifest file as shown below.

Getting started with GKE Page 30 of 44

Serving multiple applications on a Load Balancer

Getting started with GKE Page 31 of 44

Understanding ConfigMaps
ConfigMaps are Kubernetes objects for injecting containers with configuration data. ConfigMaps
keep containers agnostic of Kubernetes. They can be used to store individual properties , entire
configuration file or may be json files. ConfigMaps hold configuration in key-value pairs accessible
to PODS at run time. They are typically exposed to the containers and pods through dedicated
volume. ConfigMaps are API object similar to replica object or deployment.

Sample manifest file for configMaps

apiVersion: v1
kind: ConfigMap
metadata:
name: names-config
namespace: default
data:
employee.name: 'Renu Sood’

Configmaps can also be created directly using kubectl command as below

Kubectl create configMap demo –from-literal=‘employee.name=Renu Sood’
Once ConfigMaps are defined you, you need to associate them with your deployments.
There are two ways use configmaps in deployments:

 Add them using env

 Add them as volumeMounts

Getting started with GKE Page 32 of 44

  Adding Configmaps using env

Getting started with GKE Page 33 of 44

Understanding Secrets
Secret is Kubernetes API object that contains sensitive data such as password, token or key, size of
each secret cannot exceed 1 MB. Secrets are registered with Kubernetes master.Secrets can be
mounted as volumes or exposed as environment variables.

To create a secret, you can either by using yaml file or using from-literal option in the kubectl
command

Creating secret using KUBCETL and from-literal

Kubectl create secret test-demo –from-literal=‘password=helloworld’

Creating secret using confilg file
create secret generic jenkins --from-file=options --namespace=jenkinsgcp
Options file
--argumentsRealm.passwd.jenkins=CHANGE_ME --argumentsRealm.roles.jenkins=admin
Sample manifest file to create secret

apiVersion: v1
kind: Secret
metadata:
labels:
application: test
name: ftpserver-credentials
namespace: test-prod
stringData:
username: test
password: testpassword
type: kubernetes.io/basic-auth

apiVersion: v1
Kind: Secret
metadata:
name: mysecret
type: Opaque
data:
password: cmVudQo=
Using Secret in the pod
In the deployment.yaml , add following entry
env:

Getting started with GKE Page 34 of 44

 - name: PASSWORD
valueFrom:
secretKeyRef:
name: demo
key: password

One can verify the environment details of a pod using following command: kubectl exec -ti
<podname> env

Using Secret in the pod

Secrets can also be consumed using volume mounts
volumeMounts:
- name: secret
mountPath: /mnt

volumes:
- name: secret
secret:
secretName: secret-config

It will create file secret-config in mountPath /mnt

kubectl exec -ti demoapp-f8678586-z5nh2 env cat /mnt/secret-config

Getting started with GKE Page 35 of 44

Autoscaling at cluster level

Cluster autoscaler
Overview
GKE's cluster autoscaler automatically resizes clusters based on the demands of the workloads.
With autoscaling enabled, GKE automatically adds a new node to your cluster if you've created new
Pods that don't have enough capacity to run; conversely, if a node in your cluster is underutilized
and its Pods can be run on other nodes, GKE can delete the node.
Cluster autoscaling allows you to pay only for resources that are needed at any given moment, and
to automatically get additional resources when demand increases.
Keep in mind that when resources are deleted or moved in the course of autoscaling your cluster,
your services can experience some disruption. For example, if your service consists of a controller
with a single replica, that replica's Pod might be restarted on a different node if its current node is
deleted. Before enabling autoscaling, ensure that your services can tolerate potential disruption or
that they are designed and configured so that downscaling does not disrupt Pods that cannot be
interrupted.
How cluster autoscaler works
The cluster autoscaler works on a per-node pool basis. For each node pool, the autoscaler
periodically checks whether there are any Pods that are not being scheduled and are waiting for a
node with available resources. If such Pods exist, and the autoscaler determines that resizing a node
pool would allow the waiting Pods to be scheduled, then the autoscaler expands that node pool.
Cluster autoscaler also measures the usage of each node against the node pool's total demand for
capacity. If a node has had no new Pods scheduled on it for a set period of time, and all Pods
running on that node can be scheduled onto other nodes in the pool, the autoscaler moves the Pods
and deletes the node. If the node is not able to be drained gracefully after a timeout period
(currently 10 minutes), the node is forcibly terminated. The grace period is not configurable for
GKE clusters.
Cluster autoscaler works based on Pod resource requests, based on how many resources your Pods
have requested. Cluster autoscaler does not take into account the resources your Pods are actively
using. Essentially, cluster autoscaler trusts that the Pod resource requests you've provided are
accurate and schedules Pods on nodes based on that assumption.
If your Pods have requested too few resources (or haven't changed the defaults, which might be
insufficient) and your nodes are experiencing shortages, cluster autoscaler does not correct the
situation. You can help ensure cluster autoscaler works as accurately as possible by making explicit
resource requests for all of your workloads.

Getting started with GKE Page 36 of 44

Autoscaling a cluster
Enabling or disabling cluster autoscaling may cause the Kubernetes master to restart, which takes
several minutes to complete.
The following command creates a cluster of size 30, with node autoscaling based on cluster load
that scales the default node pool to a maximum of 50 nodes and a minimum of 15 nodes:

gcloud container clusters create [CLUSTER_NAME] --num-nodes 30 \

--enable-autoscaling --min-nodes 15 --max-nodes 50 [--zone [COMPUTE_ZONE]]
In this command:
--enable-autoscaling indicates that autoscaling is enabled.
--min-nodes specifies the minimum number of nodes for the default node pool.
--max-nodes specifies the maximum number of nodes for the default node pool.
--zone specifies the [compute zone] in which the autoscaler should create new nodes.

Adding a node pool with autoscaling

The following command creates a node pool of size 3 (default), with node autoscaling based on
cluster load that scales the node pool to a maximum of 5 nodes and a minimum of 1 node:

gcloud container node-pools create [POOL_NAME] --cluster [CLUSTER_NAME] \

--enable-autoscaling --min-nodes 1 --max-nodes 5 [--zone [COMPUTE_ZONE]
In this command:
--cluster indicates the cluster in which the node is created.
--enable-autoscaling indicates that autoscaling is enabled.
--min-nodes specifies the minimum number of nodes for the node pool.
--max-nodes specifies the maximum number of nodes for the node pool.
--zone specifies the [compute zone] in which the autoscaler should create new nodes.

Enabling autoscaling for an existing node pool

Getting started with GKE Page 37 of 44

gcloud container clusters update [CLUSTER_NAME] --enable-autoscaling \
--min-nodes 1 --max-nodes 10 --zone [COMPUTE_ZONE] --node-pool default-pool
Disabling autoscaling for an existing node pool

gcloud container clusters update [CLUSTER_NAME] --no-enable-autoscaling \

--node-pool [POOL_NAME] [--zone [COMPUTE_ZONE] --project [PROJECT_ID]]

Autoscaling PODS/Deployments

Getting started with GKE Page 38 of 44

There are two types of Autoscaling at pod level: Vertical, Horizontal

Vertical Scaling: Vertical scaling refers to adding more resources (CPU/RAM/DISK) to your server
(database or application server is still remaining one) as on demand.
Horizontal Scaling: Horizontal Scaling is a must use technology – whenever a high availability of
(server) services are required. Scaling horizontally involves adding more processing units or
phyiscal machines to your server or database. It involves growing the number of nodes in the
cluster, reducing the responsibilities of each member node by spreading the key space wider and
providing additional end-points for client connections.
Horizontal Scaling has been historically much more used for high level of computing and for
application and services.
Autoscaling in Kubernetes has two dimensions
Cluster Autoscaling: which deals with node scaling operations
Horizontal Pod Autoscaler: which automatically scales the number of pods in a deployment or
replica set
Cluster Autoscaling, together with the Horizontal Pod Autoscaler, can be used to dynamically adjust
computing power, as well as the level of parallelism that your system needs to meet SLAs. While
Cluster Autoscaling is highly dependent on the underlying capabilities of the cloud provider that’s
hosting your cluster, the HPA operates independently of your IaaS/PaaS provider.
The HPA is included in Kubernetes by default, and is implemented as a control loop, with a period
controlled by the controller manager’s --horizontal-pod-autoscaler-sync-period flag, whose default
value is 30 seconds. The controller manager queries resource utilization against the metrics
specified in each HorizontalPodAutoscaler definition.

There are three kinds of metrics:

per-pod resource metrics - like cpu, memory whose metrics are fetched from the resource metrics
API for each pod targeted by the HorizontalPodAutoscaler, then compared against a
targetAverageUtilization value or a raw targetAverageValue value.
per-pod custom metrics - like per-pod resource metrics, but which are fetched from the custom
metrics API and for who you can’t specify targetAverageUtilization values, only raw
targetAverageValue values.
object metrics - a single metric that is fetched (which describes the object in question), and
compares it to the target value

Getting started with GKE Page 39 of 44

Metrics are fetched either from the resource metrics API (for per-pod resource metrics), or custom
ones from Custom Metrics API .The Custom Metrics API, along with the aggregation layer, make it
possible for monitoring systems like Prometheus to expose application-specific metrics to the HPA
controller.

Prometheus is an open-source software application used for event monitoring and alerting. It
records real-time metrics in a time series database (allowing for high dimensionality) built using a
HTTP pull model, with flexible queries and real-time alerting.
A Pipeline deployment will automatically install and dynamically configure Prometheus to collect
metrics in a centralized (or federated Prometheus) cluster

To Autoscale your deployments, you need to create HorizontalPodAutoscaler resource.

Example of HPA based on average CPU and memory usage:
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: demo-example
spec:
scaleTargetRef:
apiVersion: extensions/v1beta1
kind: Deployment
name: test
minReplicas: 1
maxReplicas: 5
metrics:
- type: Resource
resource:
name: cpu
targetAverageUtilization: 80
- type: Resource
resource:

Getting started with GKE Page 40 of 44

 name: memory
targetAverageValue: 1000Mi

Example of Autoscaling based on pubsub

Step 1: Deploy Custom Metrics Stackdriver Adapter
To grant GKE objects access to metrics stored in Stackdriver, you need to deploy the Custom
Metrics Stackdriver Adapter. In order to run Custom Metrics Adapter you must grant your user the
ability to create required authorization roles by running the following Kubernetes command:
kubectl create clusterrolebinding cluster-admin-binding --clusterrole cluster-admin --user "$
(gcloud config get-value account)"
To deploy the adapter in your cluster, run the following command:
kubectl create -f https://raw.githubusercontent.com/GoogleCloudPlatform/k8s-stackdriver/
master/custom-metrics-stackdriver-adapter/deploy/production/adapter.yaml

Step 2: Create HorizontalPodAutoscaler object

Once you have deployed Custom Metrics Stackdriver Adapter, you can deploy a
HorizontalPodAutoscaler to autoscale your Deployment.
Apply following yaml kubectl apply -f hpa.yaml
Contents of hpa.yaml
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: pubsub
spec:
minReplicas: 1
maxReplicas: 5
metrics:
- external:
metricName: pubsub.googleapis.com|subscription|num_undelivered_messages
metricSelector:
matchLabels:
resource.labels.subscription_id: echo-read
targetAverageValue: "2"
type: External
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: pubsub

Getting started with GKE Page 41 of 44

Apply following yaml kubectl apply -f hpa.yaml

Vertical Autoscaling
You can use the VerticalPodAutoscaler custom resource to analyze and adjust your containers' CPU
requests and memory requests. You can configure a VerticalPodAutoscaler to make
recommendations for CPU and memory requests, or you can configure it to make automatic
changes to your CPU and memory requests.
Enabling vertical pod autoscaling for a cluster:
gcloud beta container clusters create [CLUSTER_NAME] --enable-vertical-pod-autoscaling --cluster-
version=1.11.8

Getting started with GKE Page 42 of 44

to enable vertical pod autoscaling for an existing cluster, enter this command:
gcloud beta container clusters update [CLUSTER-NAME] --enable-vertical-pod-autoscaling

Vertical Autoscaling

kubectl create -f my-rec-vpa.yaml

kubectl create -f my-rec-vpa.yaml

Getting started with GKE Page 43 of 44

 Appendix

 Guidelines for creating scalable

clusters :https://cloud.google.com/kubernetes-engine/docs/concepts/scalability

 Autoscaling : Reference link https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-

autoscaler

 Complete example of PUB/SUB based auto scaling: https://cloud.google.com/kubernetes-

engine/docs/tutorials/external-metrics-autoscaling

 Introduction to Jenkins plugin : https://cloud.google.com/blog/products/devops-sre/introducing-

the-jenkins-gke-plugin-deploy-software-to-your-kubernetes-clusters?
utm_source=facebook&utm_medium=unpaidsocial&utm_campaign=gcpcloud-
hybridplatformmodernization-
notrelevant&utm_content=announcementoracquisition&linkId=69798512

 Git hub Link to manifest files used in this document : https://github.ibm.com/gcp-assets/reference-

manifest-files

 Git hub link for reference code used in this document : https://github.ibm.com/gcp-assets/gke-
jenkins-pipeline

 Kubernetes Best practices Organizing with Namespaces :

https://cloud.google.com/blog/products/gcp/kubernetes-best-practices-organizing-with-
namespaces

 Kubernetes Troubleshooting a pod :

https://managedkube.com/kubernetes/pod/failure/crashloopbackoff/k8sbot/troubleshooting/
2019/02/12/pod-failure-crashloopbackoff.html

Getting started with GKE Page 44 of 44