A CIOs Guide to

Kubernetes in
Production
replex.io 1
A CIOs Guide to Kubernetes in Production

Topics
Monitoring 1

High Availability, Backup and Disaster Recovery 2

Distributed DevOps and SRE 4

CI/CD 5

Choosing the Right Kubernetes Distribution 7

Storage 9

Networking 10

Security, Identity and Access Control 13

Cost Management 16

Application lifecycle 17
Here is our complete guide to Kubernetes in Production for
CIOs and CTOs. The guide covers the topics of Monitoring,
High availability, storage, networking, security and access
control, cost management, CI/CD, application lifecycle,
distributed DevOps and SRE and choosing a Kubernetes
distribution.
Monitoring
Monitoring for Cloud-Native Applications
The cloud-native set of tools have changed the way software is
developed, deployed and managed. This new toolset has
necessitated a shift in the way both the tools themselves as well
as the applications propped up by them are monitored.
The same is true of Kubernetes, which introduces a number of
new abstractions on both the hardware as well as the application
layer. Any monitoring pipeline for Kubernetes needs to take both
these new abstractions as well as its resource management
model into account.
This means that in addition to monitoring historically relevant
infrastructure metrics like CPU and RAM utilization for cloud VMs
replex.io
and physical machines, logical abstractions like pods, services
and replica sets also need to be considered.
Observability Paradigm for Kubernetes
More importantly, however, Kubernetes monitoring needs to pivot
to a new observability paradigm. Traditionally organizations have
relied on black box monitoring methods to monitor infrastructure
and applications. Black box monitoring observes only the external
behavior of a system.
In the cloud-native age of containers, orchestration, and
microservices, monitoring needs to move beyond black box
monitoring. Black box monitoring can still serve as the baseline for
a monitoring strategy but it needs to be complemented by newer
white box monitoring methods more suited to the distributed,
ephemeral nature of containers and Kubernetes.
Observability encompasses both traditional black box monitoring
methods in addition to newer monitoring paradigms like logging,
tracing and metrics (together known as white box monitoring).
Observability pipelines decouple data collection from data
ingestion by introducing a buffer.
1
A CIOs Guide to Kubernetes in Production
The pipeline serves as the central repository of traces, metrics,
logs and events which are then forwarded to the appropriate
service using a data router. This mitigates the need to have
agents for each destination running on each host and reduces the
number of integrations that need to be maintained. It also allows
enterprises to avoid vendor lock-in and quickly test new SaaS-
based monitoring services.
Observability Best Practices
Observability aims to understand the internals of a system and
how it works to quickly debug and resolve issues in production.
Since it integrates logs, traces and metrics into traditional
monitoring pipelines it covers much more ground and requires a
lot more effort to deploy.
A best practice, therefore, is for CIOs and CTOs to gradually build
towards a full observability pipeline for their cloud-native
environments by integrating elements of white box monitoring over
time.
The adoption of cloud-native technologies has also resulted in
much more overlap between traditional dev and ops teams.
replex.io
Observability pipelines allow organizations to better integrate
these teams by helping build a culture based on facts and
feedback.
High Availability, Backup and Disaster
Recovery
High availability and disaster recovery are crucial elements of any
enterprise application. Orchestration engines like Kubernetes
introduce additional layers which have to be considered when
designing highly available architectures.
Multi-Layered High Availability
Highly available Kubernetes environments can be seen in terms of
two distinct layers or levels. The bottom-most layer is the
infrastructure layer, which can refer to any number of public cloud
providers or physical infrastructure in a data center. Next is the
orchestration layer which includes both hardware and software
abstractions like nodes, clusters, containers and pods as well as
other application components.
2
A CIOs Guide to Kubernetes in Production
High Availability on the IAAS and On-premises Layer
Public cloud providers provide a number of high availability
mechanisms for compute, storage and networking that should
serve as a baseline for any Kubernetes environment. CIOs and
CTOs also need to bake in redundancy into compute, storage and
networking equipment supporting Kubernetes environments in on-
premise data centers.
High Availability on the Orchestration Layer
On the orchestration layer, a multi-master Kubernetes cluster is a
good starting point. Master nodes should also be distributed
across cloud provider zones to ensure they are not affected by
outages in any one zone.
Availability on the orchestration layer, however, needs to move
beyond simple multi-master clusters. A best practice is to
provision a minimum of 3 master nodes distributed across multiple
zones. Similarly, worker nodes should also be distributed across
zones for high availability.
In addition to having at least 3 master nodes, a best practice is to
replicate the etcd master component and place it on dedicated
replex.io
nodes. It is recommended to have at least 5 etcd members for
production clusters.
On the application layer, CIOs and CTOs need to ensure the use
of native Kubernetes controllers like statefulsets or deployments.
These will ensure that the desired number of pod replicas are
always up and running.
Backup and Disaster Recovery
Backup and disaster recovery should also figure at the top of
every CIOs to-do list for Kubernetes clusters in production. The
etcd master component is responsible for storing the cluster state
and configuration. Having a plan for regular etcd backups is,
therefore, a best practice. Stateful workloads on Kubernetes
leverage persistent volumes which also need to be backed up.
Backup and disaster recovery are important elements of mission-
critical enterprise applications. CTOs and CIOs need to have a
well thought out and comprehensive high availability, backup and
disaster recovery mechanism for Kubernetes, that encompasses
all layers.
3
A CIOs Guide to Kubernetes in Production

In the new cloud-native world, however, the boundary between

Distributed DevOps and SRE
dev and ops has blurred even more. CIOs and CTOs need to
ensure that every DevOps team has the required skills and
The future of enterprise software is moving towards containerized
knowledge to automate, monitor and optimize the distributed,
microservices based distributed applications deployed on
cloud-native applications being developed. They should also have
Kubernetes with the cloud as an underlying layer. This new cloud-
the required skills to ensure highly available and scalable
native landscape needs to be reflected in the way dev and ops
applications, implement networking as well as onboard the tools
teams are organized internally as well as in the software release
required throughout the application lifecycle.
cycle.

DevOps and SRE

Cloud-Native Roles and Teams
One way to inject these skills into already existing DevOps teams
Kubernetes and cloud-native technologies have changed
is to move towards SRE. SRE is an implementation of DevOps,
traditional dev and ops roles, broken down siloed dev and ops
developed internally by Google that pushes for an even more
teams as well as changing the entire software release lifecycle.
overlapping skill set for individual developers. SREs typically
Given these paradigm changes, we will outline best practices for
divide their time equally between development and ops
CIOs and CTOs in terms of role definitions, team composition and
responsibilities.
new paradigms for developing and deploying software.

In the context of Kubernetes, a best practice for CIOs and CTOs is

DevOps has already broken up the siloed development, testing
to sprinkle SREs among DevOps teams. These SREs would, in
and operations teams that serviced traditional monolithic
turn, be responsible for both development as well as managing
applications. More and more developer teams are internalizing
performance, on-boarding tools, building in automation and
ops skills.
monitoring.

replex.io 4
A CIOs Guide to Kubernetes in Production

The Role of Central IT
The increasingly distributed nature of enterprise applications
translating into distributed DevOps teams, however, does not
mean that central IT loses its significance. There does need to be
some degree of control and oversight over these teams.
can potentially lead to wastage and inefficient resource usage. A
strong central IT team will be able to govern these distributed
teams and avoid the fallouts from self-service and ballooning
resources. They will also be able to hold teams accountable.

CI/CD
Even though organizations increasingly prefer developers with
cross-domain knowledge of ops, overlapping skills do tend to
In the same way that Kubernetes and the wider cloud-native
dilute both development and ops.
technology toolset made CIOs rethink traditional dev and ops
roles, it has also required a new way of thinking about build and
A best practice, therefore, is to have a central IT team that
release cycles. Containerized, microservices based applications,
includes personnel with ops and infrastructure skill sets. This skill
developed, deployed and managed by distributed teams, are not
set will enable central IT to provide DevOps teams with critical
very suited to traditional one-dimensional build and release
services that are shared by those teams. It will also ensure that
pipelines.
organizations avoid wasted effort due to distributed teams figuring
out solutions to shared problems.
CI/CD for Distributed Teams
A best practice for CIOs, therefore, is to support distributed teams
Both the cloud and Kubernetes itself have made it increasingly
with a well-tooled and thought-out CI/CD pipeline. A robust CI/CD
easier for teams to provision and consume resources. The cloud-
pipeline is essential to fully realizing the benefits of faster release
native movement and DevOps also emphasize on agility and the
cycles and agility promised by Kubernetes and cloud-native
ability to self-service resources. This can at times lead to an
technologies. There are a number of tools that CIOs and CTOs
explosion in the number of compute resources provisioned and

replex.io 5
A CIOs Guide to Kubernetes in Production
can use to deploy CI/CD pipelines. These include Jenkins,
TravisCI, GitLab CI and Spinnaker.
Continuous Integration (CI)
CI/CD is a broad concept and touches on aspects of development,
testing and operations. When deploying a CI/CD pipeline from
scratch a good place to start is with the developer team.
Continuous integration is a subset of CI/CD that aims to increase
the frequency of code merges and automate build and test
processes.
Instead of developing new features in isolation, developers are
encouraged to merge code into the main pipeline as frequently as
possible. An automated build is created from these code changes
which is then run through a suite of automated tests. Getting
developer teams to adopt CI best practices will ensure that code
changes and new features are always ready to be pushed out to
production.
Continuous Delivery and Deployment (CD)
Once CI practices are firmly in place, CIOs and CTOs can then
move on to continuous delivery and deployment. Continuous
replex.io
delivery is an extension of continuous integration where code
changes are run through more rigorous tests and ultimately
deployed to an environment that closely mirrors the production
environment.
With continuous delivery there is often a human element involved
making decisions about when and how frequently to push code
into production. Continuous deployment automates the entire
pipeline by automatically pushing code into production once it
passes the automated builds and tests defined in both the
integration and delivery phases.
CI/CD Best Practices
Agile distributed teams working in isolation can at times lead to an
explosion in the number of isolated build pipelines. To avoid this, a
best practice for CIOs is to make the CI/CD pipeline the only way
to push code into production. This will ensure that all code
changes are pushed into a unified build pipeline and are subjected
to a consistent set of integration and test suites.
Distributed teams also tend to use a number of different tools and
frameworks. CIOs need to ensure that the CICD pipeline is flexible
enough to accommodate this usage.
6
A CIOs Guide to Kubernetes in Production
Another best practice is to encourage a culture of small
incremental code changes and frequent merges among developer
teams. Smaller changes are easier to integrate and roll back and
minimize the fallout if something goes wrong.
CIOs also need to institute a build once policy at the start of the
pipeline. This ensures that later phases of the CI/CD pipeline have
a consistent build to work with. It also avoids any inconsistencies
that can creep in when using multiple build tools.
Additionally, CIOs need to strike a balance between the extent of
the testing regime they push code changes through and the speed
of the pipeline itself. More rigorous testing regimes while
minimizing the chances of bad code being pushed to production
also have a time overhead.
CI/CD pipelines even though championing decentralization and
agility do still need to be governed by central IT for major feature
releases. CIOs and CTOs need to ensure they strike a balance
between governance and oversight from central IT and the agility
and flexibility of distributed teams. They need to ensure a degree
replex.io
of oversight that while allowing them control does not impact the
release velocity of software and teams.
Choosing the Right Kubernetes
Distribution
Even though Kubernetes on its own is vastly feature rich, mission-
critical enterprise workloads need to be supported by more feature
rich variants to provide required service levels.
Managed Kubernetes
There are a number of managed Kubernetes offerings from public
cloud providers that CIOs and CTOs can evaluate. These
managed offerings take over some of the heavy lifting involved in
managing upgrades, patches and HA.
Public cloud provider offerings do, however, restrict Kubernetes
environments to a specific vendor and might not fit well with a
future hybrid or multi-cloud strategy.
7
A CIOs Guide to Kubernetes in Production
Commercial value-added Kubernetes distributions are also
available from vendors like Red Hat, Docker, Heptia, Pivotal and
Rancher. Below we will outline some of the features CIOs and
CTOs need to look for when choosing one.
Feature Set for Kubernetes Distributions
High availability and disaster recovery: CIOs and CTOs need
to look for distributions that support high availability out of the box.
This would include support for multi-master architectures, highly
available etcd components as well as backup and recovery.
Hybrid and multi-cloud support: Vendor lock-in is a very real
concern for the modern enterprise. To ensure Kubernetes
environments are portable, CIOs need to choose distributions that
support a wide range of deployment models, from on-premise to
hybrid and multi-cloud. Support for creating and managing
multiple clusters is another feature that should be evaluated.
Management, upgrades and Operational support: Managed
Kubernetes offerings also need to be evaluated based on ease of
setup, installation, and cluster creation as well as day 2 operations
including upgrades, monitoring and troubleshooting. A baseline
replex.io
requirement should be support for fully automated cluster
upgrades with zero downtime. The solution chosen should also
allow upgrades to be triggered manually. Monitoring, health
checks, cluster and node metrics and alerts and notifications
should also be a standard part.
Identity and access management: Identity and access
management are important both in terms of security as well as
governance. CIOs need to ensure that the Kubernetes distribution
they choose supports integration with already existing
authentication and authorization tools being used internally. RBAC
and granular access control are also important feature sets that
should be supported.
Networking and Storage: The Kubernetes networking model is
highly configurable and can be implemented using a number of
options. The distribution chosen should either have a native
software-defined networking solution that covers the wide range of
requirements imposed by different applications or infrastructure or
support one of the more popular CNI based networking
implementations including Flannel, Calico, kube-router or OVN
etc. CIOs also need to ensure that the Kubernetes distribution
8
A CIOs Guide to Kubernetes in Production
they choose supports at a minimum, either flexvolume or CSI
integration with storage providers as well as deployment on
multiple cloud providers and on-premise.
Deploy, manage and upgrade applications: Kubernetes
distributions being considered by CIOs also need to support a
comprehensive solution for deploying, managing, and upgrading
applications. A helm based, application catalog that aggregates
both private and public chart repositories should be a minimal
requirement.
Storage
Kubernetes storage is a hard nut to crack. Kubernetes was initially
designed to support stateless applications that do not use saved
data across sessions. Kubernetes pods are meant to be
ephemeral and are constantly created, destroyed and moved
across nodes. Whenever pods are destroyed Kubernetes volumes
attached to these pods are also terminated.
replex.io
Most legacy applications, including databases, however, are
stateful and store tons of data for use across sessions. Since
volumes can only be used to store temporary data, they are not
well suited to these applications.
Kubernetes Persistent Volumes
In order to support stateful applications, Kubernetes introduced a
new type of volume plugin called persistent volumes. The
persistent volume resource decouples storage from the pod
lifecycle and allows data to persist across pod restarts making
them a good candidate for stateful applications.
Software-Defined Storage (SDS)
Software-defined storage (SDS) solutions are a good bet for CIOs
and CTOs looking for a Kubernetes storage solution. Kubernetes
supports a number of these SDS providers as persistent volume
plugins (StorageOS, CephFS, Portworx, GlusterFS, ScaleIO and
Quobyte).
SDS solutions abstract storage from the underlying hardware and
present it for consumption as shared storage pools. SDS solutions
also abstract away the complexities of having to manage
9
A CIOs Guide to Kubernetes in Production
disparate storage devices and filesystem types. Built-in APIs allow
consumers to manage and automate encryption, high availability,
backups and replication.
Storage Best Practices
A best practice when choosing a storage solution for Kubernetes
is to ensure it is distributed, resilient, durable and robust with no
single points of failure. The storage solution chosen should also
support dynamic, on-demand provisioning with support for
Kubernetes storage classes and be easily scalable. Dynamic
provisioning significantly reduces the management overhead of
creating, managing and configuring persistent volumes and
making them available for consumption in the cluster. Storage can
be automatically provisioned whenever it is requested by users.
Another best practice is for CIOs and CTOs to evaluate SDS
solution based on support for high availability, automated
replication and backups.
Additional features to look for in a storage provider is support for
dynamic resizing, automated volume snapshots, backup and
restore. As with Kubernetes and the cloud-native toolset, the
replex.io
storage solution chosen should also be declarative, and cloud-
agnostic with support for automated upgrades. Encryption and
access control should also be required features.
With the emphasis on multi-cloud deployments, CIOs and CTOs
should also ensure that any storage solution they choose is
portable and cloud-agnostic. Storage solutions should be able to
pool storage resources from disparate (cloud, on-prem etc.)
hardware sources.
Performance is another aspect that needs to be considered. In
most cases, the storage solution chosen will depend on the
unique attributes of each environment and application
requirements. Before undertaking a review of SDS solution based
on the features outlined above, CIOs and CTOs should
benchmark the performance requirements of their environments
and applications.
Networking
As with storage, networking is also an important component of
enterprise environments. There are three important elements
10
A CIOs Guide to Kubernetes in Production
CIOs and CTOs need to consider when setting up networking for
a Kubernetes environment: communication between application
components (pod to pod communication), communication
between pods and services and communication with the outside
internet.
Kubernetes Networking Model
The Kubernetes networking model allots a unique IP to each
individual pod. By default, all pods belonging to a Kubernetes
cluster can communicate with all other pods. This communication
happens across Namespaces, services, or nodes.
The networking model also allows groups of pods that provide the
same functionality (Services) to communicate with other pods.
The Service abstraction de-couples groups of dependent pods
and allows applications to continue functioning in the event of pod
restarts.
Kubenet, the default networking plugin from Kubernetes provides
some basic networking functionality but is limited when it comes to
cloud environments. For a more feature rich networking solution
that can support mission critical enterprise environments, CIOs
replex.io
and CTOs should consider one of the CNI compatible networking
plugins.
Feature set for CNI Evaluation
Flannel, Calico, Canal, kube-router and Weave Net are some of
the more famous CNI plugins. Below we review some of the
features that CIOs and CTOs need to consider when choosing a
CNI for their Kubernetes environment.
Support for Network Policy
Support for Kubernetes network policies is a crucial functionality
that CIOs and CTOs should use to evaluate CNI plugins. Network
policies allow DevOps to configure and control traffic to and from
their applications. Network policies perform both a security and an
access control function.
With the network policy resource, Kubernetes enables a shift left
approach where DevOps can configure network policies using the
same concepts used to deploy applications.
By default, pods do not filter incoming traffic and there are no
firewall rules. Network policies allow granular control over how
11
A CIOs Guide to Kubernetes in Production
pods are allowed to communicate among each other and with
other network endpoints. This control extends to both ingress as
well as egress traffic.
Type of Network
When choosing a CNI plugin CIOs and CTOs should also
consider the type of network deployed. Plugins that deploy an
overlay network, usually encapsulate packets in an extra layer.
Flannel for example deploys a layer 3 overlay network and uses a
backend like VXlan, IPsec or host-gw for packet forwarding. Since
overlay networks encapsulate packets in an extra layer, they
increase the network overhead and can affect performance.
Additionally, tracing network packets is often difficult with plugins
that encapsulate traffic.
Calico in contrast deploys a Layer 3 network that uses BGP to
route traffic between hosts. Using BGP has obvious performance
benefits since packets do not have to be encapsulated using a
backend. It also makes troubleshooting faster and easier using
already existing tools. For situations where an overlay network is
needed like routing traffic between AZs, it does use encapsulation
like IP-in-IP or VXlan.
replex.io
Encryption
Encryption is another feature to look out for especially in
environments that exchange traffic across untrusted networks.
The Weave works CNI plugin transports data using the fastest
available method and also encrypts it: TCP and UDP traffic for the
sleeve method is encrypted using NACL encryption library, data
plane traffic for the fast datapath method is encrypted using ESP
of IPsec.
Support for Service Mesh
CIOs and CTOs looking for more control over networking, security
and access should also ensure that the network plugin chosen
supports integration with service meshes like Istio. This is also
important given the sheer scale and complicated nature of service
to service communications for cloud-native applications.
Service, Load Balancing and Ingress
CNIs alone do not address how the cluster communicates with the
internet. There are a number of native Kubernetes abstractions
that allow traffic to be exchanged with the wider internet. The
service abstraction is one, Kubernetes ingress is another.
12
A CIOs Guide to Kubernetes in Production
ClusterIP, NodePort, LoadBalancer and ExternalName are all
service types that allow external traffic into the cluster.
LoadBalancer service type is the standard way to expose services
to the internet. It does however require a supported cloud provider
to be present and can get expensive since each exposed service
gets its own IP address.
For most enterprise environments, Kubernetes Ingress is the
recommended method to expose services to the internet. Ingress
handles load balancing at Layer 7 and officially supports nginx
and GCE ingress controllers. There are a number of additional
Ingress controllers including Contour and Istio that CIOs and
CTOs can look into. Ingress is more feature rich as compared to
the LoadBalancer service type and is also a less expensive
option.
Security, Identity and Access Control
Kubernetes security extends beyond the immediate cluster
environment to applications and infrastructure. CIOs and CTOs
replex.io
need to ensure that security is a part of the entire application
lifecycle and encompasses all layers.
Access Control
With the cloud-native movement, identity and access control have
become increasingly important in the context of security. Native
Kubernetes authentication, authorization and admission
controllers allow CIOs and CTOs to draw a security perimeter
around their environments, identify users and processes and
govern the resources they are allowed to access.
There are two ways requests can be authenticated in Kubernetes:
normal accounts and service accounts. Normal accounts usually
correspond to user accounts and are managed by an outside
third-party service. A best practice for CIOs and CTOs is to enable
multiple authentication methods: one each for user accounts
(either OpenID Connect or X509 Client Certificates) and service
accounts (Service account tokens).
Kubernetes RBAC
Once requests are authenticated, they can then be granted
permission to access via RBAC. RBAC allows CIOs and CTOs to
13
A CIOs Guide to Kubernetes in Production
regulate and govern access to resources based on roles of
individual users.
Cluster roles grant permissions for the entire cluster across all
Namespaces. CIOs and CTOs should ensure that Cluster roles
are only granted to trusted users or groups of users. The cluster-
admin role specifically, has a very wide range of permissions to
perform actions and has access to all resources. A best practice
therefore, is to avoid granting the cluster-admin role as much as
possible.
Roles are by default restricted to specific Namespaces and should
be preferred over Cluster Roles, whenever possible. For this to
work a best practice is to wall off groups of resources into
individual namespaces for teams, departments, clients, and
applications etc. Roles can then be used to implement fine-
grained access control by specifying the apiGroup of the resource,
the resource itself (e.g. pods) and the operations that can be
performed. Roles are granted to individual users or groups of
users using Role Bindings.
replex.io
A best practice in the context of RBAC is to follow user-access
best practices and keep the scope of permissions small. CIOs and
CTOs however do need to consider the increased management
overhead that comes with a fine-grained RBAC policy.
Continuous Security Scanning
CIOs and CTOs also need to encourage a culture of periodic
security scanning of container images. This can be accomplished
using tools like Claire and Anchore. Periodic scanning will identify
any common security vulnerabilities (CVEs) in container images.
To make this a continuous process, a best practice is to bake in
security and vulnerability checks and tools into the CI/CD pipeline.
Any new code as well as the container images built using this
code should be checked for CVEs as part of the CI/CD pipeline.
CIOs and CTOs should also discourage the use of unknown
images from public repositories and prioritize the use of private
registries.
The AlwaysPullImages admission controller is another way to
ensure images are always pulled with the correct authorization
and cannot be reused by pods.
14
A CIOs Guide to Kubernetes in Production
Network Segmentation
By default, Kubernetes pods and containers are allowed to
communicate across nodes, namespaces and services. To reduce
the potential attack surface, limit the area of impact and fallout a
best practice is to segment these communications.
CIOs and CTOs can use the native Kubernetes network policy
resource to control and specify how pods or groups of pods are
allowed to communicate. A network policy however only works in
tandem with a network plugin. CIOs and CTOs should ensure that
the networking solution they choose supports a network policy.
Kubernetes Secrets
Kubernetes has a native secret resource that allows sensitive
information like passwords, tokens or keys to be stored. It is
recommended best practice to store such sensitive information in
secrets rather that directly specifying them in pods or containers
spec. CIOs and CTOs should also ensure that all secrets are
encrypted at rest. They can do this using any of the four providers
aescbc, secretbox, aesgcm or kms. Data should also be
encrypted in transit using one of the supported network plugins.
replex.io
Logging
Kubernetes environments are complex with many different
components and layers. Logs are a good way to keep tabs on
cluster activity, understand what is happening and debug
problems. CIOs and CTOs should put considerable thought into
designing a logging architecture for Kubernetes.
The native logging functionality for Kubernetes only stores logs for
the lifetime of a container, pod or node. A best practice therefore
is to use a cluster level logging architecture and configure a
separate backend for storing Kubernetes logs.
Auditing
Kubernetes auditing is another native resource that can help CIOs
and CTOs gain more security insights into cluster activities. Audits
keep track of what events happened inside the cluster, who
performed those activities and which resources they affected.
Rules about which events to record and the associated metadata
can be defined in the audit policy resource.
CIOs and CTOs should also avoid giving direct access to
Kubernetes nodes.
15
A CIOs Guide to Kubernetes in Production
Cost Management
Freedom, agility, self-service and the ability to move fast are core
principles of modern DevOps. They are also baked-in into most
cloud-native tooling including Kubernetes. These concepts allow
distributed DevOps teams to provision and consume resources
with minimal oversight from central IT. This can at times lead to
increased wastage of resources and ballooning costs. CIOs and
CTOs therefore need to implement a comprehensive resource
governance and cost management framework to control costs.
Resource Governance using Namespaces
Kubernetes namespaces are a great way to implement low level
resource governance mechanisms. Creating separate
namespaces for teams will allow CIOs to control the resource
consumption of individual containers as well as the total resource
consumption of all containers that belong to the namespace. They
can also ensure that all containers run with default limits and
control the total number of pods or other Kubernetes objects
allowed to run.
replex.io
Cluster Autoscaler
Using a cluster autoscaler is another best practice in the context
of resource governance and cost management. The cluster
autoscaler right sizes Kubernetes clusters based on utilization
metrics of individual nodes. Nodes seeing sustained low utilization
are taken out of the cluster pool by the cluster autoscaler.
Trimming down clusters to reduce resource wastage is a good
way to reduce cloud provider bills and ensure efficient resource
usage. Another good way to control costs is to right size Nodes
that see sustained levels of low utilization. This will ensure that the
resource footprint of the node matches that required by the pods
running on top.
Horizontal and Vertical Pod Autoscaler
CIOs and CTOs should also ensure the use of both the horizontal
(HPA) and vertical pod auto scalers (VPA) to govern resource
usage and cost management.
The HPA is a native Kubernetes resource that increases or
decreases the number of pod replicas for an application based on
16
A CIOs Guide to Kubernetes in Production

CPU utilization metrics. HPA ensures that controllers are right

Application lifecycle
sized and do not result in resource wastage.

We have already looked at CI/CD tooling in the context of

Similarly, the VPA ensures that individual pods use resources
Kubernetes and how it allows CIOs and CTOs to accelerate
efficiently. It continuously monitors and recommends optimal
software delivery and shorten release cycles. In adopting
values for resource requests, kills the pods that do not have the
Kubernetes and cloud-native, CIOs and CTOs also need to
correct values and sets the correct value when they are recreated
internalize a broader set of concepts, tools and best practices.
by the controller.

Infrastructure as code (IAC), Environment as code (EC),

Spot Instances
Immutable infrastructure (II), Declarative Configuration,
Spot instances have emerged as a viable cost management
Observability, and Operations by Pull Request are just some of
alternative to on-demand and reserved cloud provider instances.
these concepts. Integrating these concepts helps make both
On Kubernetes too, CIOs and CTOs can consider using spot
infrastructure and applications reproducible, consistent and
instances to reduce cluster costs. A best practice is to create a
traceable. It also helps build in automation, further accelerating
cluster with a mix of on-demand, reserved and spot instances that
software delivery and making it easier for DevOps teams to
is both reliable and reduces overall cluster costs. CIOs and CTOs
conduct operations.
should also develop a comprehensive tagging strategy to keep
track of resources provisioned by distributed teams. This will also
GitOps
help rapidly identify and decommission underutilized or unused
GitOps developed at Weaveworks is one such methodology that
assets.
allows CIOs and CTOs to stitch together tools and best practices
from these concepts. The GitOps workflow encompasses the
entire build, deploy, manage and monitor lifecycle.

replex.io 17
A CIOs Guide to Kubernetes in Production
Version Control
The most important cog in the GitOps workflow is a version control
system like Git. Git stores the desired system (clusters,
applications and infrastructure) state as declarative version-
controlled configuration files in Git. Any changes to production
environments happen via Git commits.
Kubernetes, like most modern cloud-native tools, is declarative in
nature and as such is perfectly suited to a GitOps workflow that
treats declarative configuration files stored in Git as the single
source of truth.
A minimal GitOps workflow includes Git for version control, a CI
tool for unit and integration tests, a private image repository, and a
deployment and release automation tool like Flux.
The workflow starts by pushing code changes to Git, making a
Pull request and reviewing and merging the code. Once code is
merged the CI tool runs the changes through an automated unit
and integration test suite, builds a new image and deposits it to a
registry. The Flux tool, which continuously monitors the registry,
replex.io
detects those updates and automatically deploys them to the
Kubernetes cluster.
Unlike regular CD tools, GitOps also compares the actual state of
the production system with the one under version control and
sends out alerts whenever it discovers a divergence. It also
triggers a convergence mechanism that brings the observed and
desired states into sync.
Environment as Code
Another GitOps best practice is to adopt a broader “Environment
as code” approach. Environment as code extends version control
to Kubernetes clusters, infrastructure and observability tooling.
Configuration files for clusters, infrastructure and observability
tooling are version controlled in Git. Since the entire system
(clusters, application, infrastructure and observability tools) is
version controlled, it is consistent and can be easily reproduced
enabling faster disaster recovery as well as making rollouts and
rollbacks more seamless.
18
A CIOs Guide to Kubernetes in Production

Audit and post mortems are also easier with the Git log serving as
an audit trail.
GitOps also facilitates DevOps and SRE teams by making
development and operations activities part of the same workflow.
Developers can easily internalize operation tasks using the
version-controlled observability stack, conducting operations tasks
and fixing production issues as pull requests rather than making
changes to the running system.
GitOps Best Practices
Below we will review some of the best practices that enable a
GitOps workflow:
• Use automated difference tools to monitor divergence
between the actual production system and the one under
version control
• Use IAC tools (Terraform, Ansible) to create server
configuration files and keep them under version control
• Use the principles of immutable infrastructure, containers
and diff tools (kubediff, ansiblediff, and terradiff) to reduce
configuration drift and ensure the desired system state is
maintained
• Version control your observability stack
• Use native Kubernetes constructs for rolling updates.

• Use Git or another version control system as the single

source of truth to store version-controlled configuration
files for the entire system (cluster, infrastructure,
applications)
• Ensure any changes to the production system are
conducted via pull requests

replex.io 19
 Get in touch
replex.io | sales@replex.io

AUTHOR

Hasham Haider
Fan of all things cloud, containers and micro-services!

*The information provided within this eBook is for general informational purposes only. While we try to keep the
information up-to-date and correct, there are no representations or warranties, express or implied, about the
completeness, accuracy, reliability, suitability or availability with respect to the information, products, services, or
related graphics contained in this eBook for any purpose. Any use of this information is at your own risk.

replex.io 20