344 BROUGHT TO YOU IN PARTNERSHIP WITH

Kubernetes Multi-
CONTENTS

•  Introduction

Cluster Management
•  What Is Multi-Cluster
Management?

•  What Is Kubernetes Governance?

and Governance
•  Core Practices for Successful
Kubernetes Governance

•  Why Is Multi-Cluster Management

and Governance Important?
•  Conclusion
STEFAN THORPE
VP OF ENGINEERING, CHERRE

In today’s technology landscape, Kubernetes plays a vital role. WHAT IS MULTI-CLUSTER

Container usage is only increasing, with 86 percent of IT leaders MANAGEMENT?
prioritizing containerization for more applications. (Cloud Container Due to the performance nature of modern cloud-native applications,
Adoption In The Enterprise, 2020) Kubernetes environments need to be highly distributed.

Kubernetes is an open-source container management system Proper multi-cluster management ensures consistent operations
designed by Google that brings significant agility, automation, and across all environments, enterprise-grade security, and workload
optimization to those running containers in a DevOps environment. management.

Thanks to Kubernetes’ extensibility, it’s able to meet the most diverse Below is a visual representation of an example Kubernetes multi-
requirements and constraints in application development, and sits cluster architecture.
firmly in the prime spot as the most common open-source container
orchestration framework.

SEE FIGURE ON PAGE 3

However, enterprise organizations who are operating and serving
customers at scale need to consider how best to leverage the
complexity of Kubernetes to fully take advantage of wider geolocated
cloud features.

Kubernetes is originally designed to support workload consolidation

in a single cluster. However, there are many problem scenarios that
require a multi-cluster approach to optimize performance and results.

These can include workloads across regions, limiting outage

blast radius, compliance issues, hard multitenancy, security, and
specialized solutions.

For example, enterprise-scale organizations might leverage multiple

Kubernetes clusters spread across multiple clouds or hybrid clouds
with on-premises and with one or more cloud providers.

REFCARD | MARCH 2022 1

The Leading Independent
Kubernetes Platform
D2iQ simplifies and automates the really difficult tasks
needed for enterprise Kubernetes in production at scale,
reducing operational burden and TCO.

Get the K8s Cheat Sheet →

 REFCARD | KUBERNETES MULTI-CLUSTER MANAGEMENT AND GOVERNANCE

Figure 1

Leveraging multiple Kubernetes clusters can solve many of the
aforementioned problem scenarios, but managing multiple clusters is a
difficult task. As the cluster grows, the complexity in managing them
also increases. There are multiple operations that occur in a well-
managed Kubernetes cluster, such as adding/removing a node, securing
the cluster, upgrading it from time to time, maintaining the cluster, etc.
•  Defining the resource quotas among the clusters
•  Creating pod budget policies
•  Creating network and governance policies
•  Defining taints and toleration on the clusters
•  Scanning for risks and vulnerabilities

These operations' complex nature at scale requires a proper Kubernetes
multi-cluster management system to monitor everything happening
on all the clusters. Taking a best practice approach, SREs in operations
teams can better monitor the health of the cluster environment and
maintain application performance when multi-cluster architecture is
more uniform.
One of the important strategies for managing multiple Kubernetes
clusters in the industry is to have a proper cluster isolation pattern.
The isolation we're discussing here improves the security within an
application.
Figure 2

Kubernetes multi-cluster operations include:

•  Creating, updating, and deleting Kubernetes clusters across

different environments (data centers; private, hybrid, and public
clouds; and at the Edge)

•  Updating the control plane and compute nodes

•  Managing application life cycles across hybrid environments

•  Scaling, securing, and upgrading clusters (perhaps even

provider-independent)

•  Maintaining and updating multiple nodes

•  Searching, finding, and modifying any Kubernetes resource

•  Implementing Role-Based Access Control (RBAC) over the

clusters. (For example, an admin can have access to all clusters,
but a developer might only need access to the dev cluster.)

REFCARD | MARCH 2022 3 BROUGHT TO YOU IN PARTNERSHIP WITH

 REFCARD | KUBERNETES MULTI-CLUSTER MANAGEMENT AND GOVERNANCE

To achieve the isolation boundary, you must optimize the use of Many of the above tools share the following features for managing
Kubernetes namespaces. These will help you in defining the scope of multiple clusters with ease and visibility:
cluster usage and its resources by a team or application. Namespaces
•  Central cluster management with automatic application
are a logical way to break a cluster up into logical constructs and also
delivery
allow you to define RBAC; pod security policies; network policies (to
•  Reduced operation cost by unifying all management
separate workloads); and quotas.
•  Increased application availability by deploying across
Figure 3
distributed clusters

•  Centralized policy management and enforcement for ease of

security compliance

•  Accelerated application development to production

•  Global DNS provision to distribute traffic across multiple

clusters in a balanced way

Automation and policy management are crucial to successfully

managing multiple clusters in multiple environments. While it is not
always possible to automate everything on day one, it should remain
a top priority to automate as much of your cluster deployment and
operations as possible as you move forward.

BENEFITS OF MULTI-CLUSTER ADOPTION

Here’s an example of this isolation in practice: You may need to deploy
a three-service app. Service A and Service B should not be able to talk
to each other, yet Service C needs to talk to both A and B. So then you
could leverage the namespace and add an additional compliance level
by keeping them in different clusters.
There are many popular tools available for Kubernetes multi-cluster
management. In no particular order, these include:
•  Cluster management
−  D2iQ Kommander
−  Rancher
−  Red Hat Advanced Cluster Management
Depending on the business or product requirements, you will need to
optimize an appropriate multi-cluster strategy when considering the
following:
RISK MITIGATION
By applying a policy-based governance approach to a multi-cluster
architecture, engineers can automatically monitor and ensure security
and configuration controls meet industry compliance standards or self-
imposed corporate standards in a desired state model.
GEOLOCATIONS OR CLOUD LOCATIONS
A multi-cluster strategy can also help address the typical issues
that administrators and site reliability engineers (SREs) face when
working across a range of environments. Multi-clustering can optimize
performance across multiple data centers, private clouds, and public
clouds that run Kubernetes clusters.

•  Deployment Leveraging multi-clusters around geolocation can improve

performance for consumers as it reduces latency over regions and also
−  CloudBolt
facilitates better disaster recovery.
−  Google Anthos

−  Helm
TEAM-BASED STRUCTURE
One of the best practice approaches to working in Kubernetes is to
−  Kubespray
create smaller multi-clusters and multiple environments (development
•  Security and monitoring and production) per cluster. That way, they will all be connected to

−  Aqua each other over the organization network. The two main reasons for
such an approach are to separate concerns and limit the blast radius
−  cAdvisor
of unforeseen events. This architectural framework approach sees the
−  Falco formation of multiple Kubernetes clusters across an organization.

REFCARD | MARCH 2022 4 BROUGHT TO YOU IN PARTNERSHIP WITH

 REFCARD | KUBERNETES MULTI-CLUSTER MANAGEMENT AND GOVERNANCE

The advantage of this approach is that teams can leverage Kube features Finally, for Kubernetes governance, you should optimize policy
like service discovery across the wider organization while maintaining management. Think about how Kubernetes is going to impact your
complete control of their own resources. engineering culture and how to find the right balance for developers’
flexibility within it. Governance — with an appropriate level of flexibility
Consider your project and business needs from the outset when
— ensures that you can meet customer needs and deploy critical
thinking about adopting a multi-cluster strategy. For example, do
services in a uniform and consistent way.
you really need hard multitenancy? Hard multitenancy is an approach
that assumes all tenants are high risk and as such requires the most The two governance dimensions consist of:
stringent of responses, which will automatically require a multi-cluster
•  Policy scope: where a specific rule should be applied, enforced,
approach. After that, consider your compliance requirements and if
or verified
you have the operational capacity to handle the overheads of a multi-
•  Policy targets: what should be enforced and verified
cluster architecture.

When you think about policy targets for governance, there can be
WHAT IS KUBERNETES GOVERNANCE?
many aspects. Here, we are discussing security policies, network
Governance is about synchronizing clusters and enforcing centralized
management, access management, and image management.
policy management. Kubernetes governance refers to a set of rules
created through policies that need to be enforced on all Kube clusters. LIMIT USER ACCESS
This is a critical component for enterprises being production-ready In security policies for Kubernetes governance, it’s important to limit
and working at scale in Kubernetes. Typically, this process means users' access on the pods in the clusters. Users of the cluster should
enforcing conformance rules across Kubernetes multi-clusters as well have well-defined access depending on the privilege, which is based
as applications running in those clusters. on their role. To achieve this, implement security policies in place that
will have rules and conditions related to access and privileges. In these
It would help if you had a solid plan in place to be production-ready at
policies, define that the containers should have only read access to the
scale. Kubernetes governance may sound dull, but it’s going to pay off
file system and enforce allowPrivilegeEscalation=false in the
in the long run, especially if implemented across a large organization.
policy so that the containers and child processes cannot be subject to
Suppose you continue to grow the number of clusters in use without
privilege changes.
governance. In that case, they will exist in different pockets with
different rules and implementations, which basically translates to a
NETWORK POLICIES
huge amount of extra time and work for your teams to manage in the
Network policies play a very important role in defining which service can
near future. However, there are only a few very important components
talk to each other. Here, define which pod and service can communicate
for creating successful Kubernetes governance, and these are outlined
with each other and where the isolation must be established. This
below.
comes under the pod security in Kubernetes governance. These policies
help you control the traffic inside the Kubernetes clusters. These
CORE PRACTICES FOR SUCCESSFUL
network policies can be pod-based, namespace-based, or IP-based,
KUBERNETES GOVERNANCE
depending on your requirement for governance. In these policies,
When considering a successful Kubernetes governance strategy, the
you define entities such as spec, podSelector/namespaceSelector,
first component is ensuring that good multi-cluster management and
policyTypes, ingress (rules defined for traffic), egress, etc.
visibility are well in place. It is necessary to maintain control over how
and where clusters are provisioned and configured, as well as which
ACCESS ADMINISTRATION AND MANAGEMENT
versions of software can be used. For visibility, ideally, you should be
In access management, while configuring the RBAC (role-based access
able to centrally view and manage clusters to better optimize resources
control) policy, admins need to take the responsibility to limit access to
and troubleshoot issues. Improved management practices and better
cluster resources. There are multiple Kubernetes objects such as Role,
visibility can also save you the headache of having to deal with a number
ClusterRole, RoleBinding, and ClusterRoleBinding, which helps
of security risks and performance issues down the road.
you define access to the cluster resources in detail.

Next, you need to have an authentication and access management

IMAGE MANAGEMENT
system. Having centralized authentication and authorization is going
Leveraging public Docker images can improve speed and agility when
to help your organization in a big way. Not only will it make the process
developing applications. However, there are a lot of vulnerable Docker
of login easier, but it’ll also help track who is doing what, so you can
images out there, and using them on the production cluster can be very
ensure the right people are performing the right tasks at any given time.
risky.

REFCARD | MARCH 2022 5 BROUGHT TO YOU IN PARTNERSHIP WITH

 REFCARD | KUBERNETES MULTI-CLUSTER MANAGEMENT AND GOVERNANCE

Image management is also a part of Kubernetes governance. Any This approach will help in reducing complexity and ensure that
and all images to be used on the cluster must be audited first for any applications running on the clusters are highly available.
vulnerabilities. There are a number of approaches to vulnerability
You will also see an improvement in performance because you can
scanning — how and where you check will depend on your preferred
deploy your application on a cluster in the nearest available region
workflows. However, it is recommended that images be tested and
where your application needs to be deployed.
passed before deploying on the cluster.

Implementing governance early makes sure that everything is

Hacking activities have increased exponentially in recent years, and
working as per rules defined in the policies and the employees of the
hackers keep on finding loopholes in the system to gain access for
organization comply with it.
faulty intentions. So, be very vigilant when adopting best practices to
ensure that you only use official, clean, and verified Docker images on The key advantages of optimizing multi-cluster management and
the cluster. governance include:

These policies are a must for better Kubernetes governance. After these •  Reduced complexity
policies and rules are in place, verify that they are all enforced in the •  Enhanced cluster visibility
organization as a part of internal compliance rules and policies. Monitor
•  Better application availability
teams and projects comprehensively to ensure that everyone in the
organization follows the policies defined and do not bypass them. •  Improved monitoring and logging

•  Ensured conformance and compliance

Open Policy Agent (OPA), a CNCF project, is an excellent way to create
and enforce such policies across an enterprise at scale. As shown below, •  Reduced latency
every request will go through the OPA, and the request decision will be •  Improved disaster recovery
based on the policies defined for the Kubernetes cluster. If the request
•  Ability to deploy legacy and cloud-native application across
is policy-compliant, it will be executed. If the request is violating the
multiple cloud/hybrid environments
policies defined, it will be rejected by the OPA.
•  Enhanced cloud solutions
Figure 4
•  Standardization with built-in automation to reduce operational
costs and improve efficiency

•  Better security posture (through visibility, centralized control,

consistency, and standardization)

CONCLUSION
There are many established vendors like D2iQ, Red Hat OpenShift,
Google Anthos, Rancher, and VMware Tanzu that are capable of
supporting you in the management and governance of Kubernetes
multi-clusters at scale.

These vendor offerings provide varying levels of openness, speed

to deployment, cost of ownership characteristics, reliability, multi-
WHY IS MULTI-CLUSTER MANAGEMENT
cloud support, and the ability to help with smart (AI/ML) cloud- native
AND GOVERNANCE IMPORTANT?
applications. While any one of these vendors can help you reap the
The complexity of managing a Kubernetes cluster only increases as
benefits and solve challenges faced while working with Kubernetes at
the size of the cluster increases. Cluster sprawl can begin with multiple
scale, it’s best to do your due diligence before committing.
different application teams using it for different projects.

For those interested in following an open-source path for multi-cluster

The solution to this is identifying and implementing best practice
management and governance (perhaps to reduce OPEX costs), there
governance early and quickly to avoid major leg work in the future.
are a lot of open-source tools available. Tools like K9s, Kontena Lens,
Implement multiple smaller clusters and a single centralized
WKSctl, cAdvisor, and Weave Scope can help you overcome the basic
management system to keep track of all the clusters and optimize
challenges involved in managing and governing Kubernetes multi-
observability (for monitoring, alerting, capacity control, and cost
clusters.
management, etc.).

REFCARD | MARCH 2022 6 BROUGHT TO YOU IN PARTNERSHIP WITH


The benefits of investing in a Kubernetes management platform lie
predominantly in the cost savings an organization can experience
by making sure everything in production runs smoothly and that
applications can run at scale across the globe without any hiccups.
Optimizing such a tool can also help you reduce manual effort amongst
your engineers through the multiple automated solutions available in
their features.
Here too, the ability of any one of these vendors to help you reduce
the workforce power involved and the bandwidth required to manage
multiple Kubernetes clusters across a large-scale enterprise can vary.
Research and test before selecting.
So, when considering how to manage multi-cloud, multi-distribution
clusters, DevOps teams should take a look at these platforms as an
alternative to overcome the pain of multi-cluster management and
governance on their own.
REFERENCES
2020. Cloud Container Adoption In The Enterprise. [PDF] Forrester
Consulting. Available at: <https://www.capitalone.com/tech/cloud-
container-adoption-report/> [Accessed 30 December 2020].
REFCARD | MARCH 2022
REFCARD | KUBERNETES MULTI-CLUSTER MANAGEMENT AND GOVERNANCE
WRITTEN BY STEFAN THORPE,
HEAD OF ENGINEERING, CHERRE
Cherre is the leader in real estate data and insight.
We help companies connect and unify their
disparate real estate data so they can make faster,
smarter decisions.
Stefan is an IT professional with 20+ years management and
hands-on experience providing technical and DevOps solutions to
support strategic business objectives.
600 Park Offices Drive, Suite 300
Research Triangle Park, NC 27709
888.678.0399 | 919.678.0300
At DZone, we foster a collaborative environment that empowers developers and
tech professionals to share knowledge, build skills, and solve problems through
content, code, and community. We thoughtfully — and with intention — challenge
the status quo and value diverse perspectives so that, as one, we can inspire
positive change through technology.
Copyright © 2022 DZone, Inc. All rights reserved. No part of this publication
may be reproduced, stored in a retrieval system, or transmitted, in any form or
by means of electronic, mechanical, photocopying, or otherwise, without prior
written permission of the publisher.
7 BROUGHT TO YOU IN PARTNERSHIP WITH