Dzone Kubernetes in The Enterprise 2022 1669219453241

BROUGHT TO YOU IN PARTNERSHIP WITH
Welcome Letter
By Alan Hohn, Director of Software Strategy at Lockheed Martin
What accounts for the ongoing success of Kubernetes That's why I'm excited to introduce this year's Kubernetes
and the vibrancy of its ecosystem? I think it's the way that Trend Report. It provides survey data showing how teams
Kubernetes grew organically out of earlier successful are deploying Kubernetes, their architectures and design
abstractions, each of which allowed development teams to patterns, the tools they're using to manage applications, and
spend less time on technology and more time on capability. the pain points they're experiencing as they maintain real-
world clusters at scale.
Abstraction has been one of the main ways computing has
advanced. As we moved from toggle switches to punch cards In addition, it has expert insights that are essential for a solid
to assembly to higher-level languages, we gained the ability Kubernetes strategy, including how to build security and
to declare desired behavior in a way that's closer to the way observability into a cluster and how to extend that security
we specify the problem. and observability to workloads that run on the cluster,
simplifying maintenance and monitoring.
As we started using virtual memory, multiprocessing, file
systems, software-defined networks, and containers, we Welcome to the 2022 Kubernetes in the Enterprise Trend
gained the ability to think of computing resources in ideal Report! I hope you enjoy reading it and gain insights that
terms — as continuous rather than discrete quantities. help you build and sustain your Kubernetes deployments
Kubernetes provides our ultimate expression so far of this so you can focus your energy on building exciting software
move toward declarative, ideal computing. on top of this amazing platform.
So why can it be so difficult to adopt and gain full advantage Best regards,
of Kubernetes? It's especially puzzling when you realize
that the core Kubernetes APIs and resources have been
quite mature these past few years. While the Kubernetes
ecosystem continues to change quickly, we now have a
stable core on which to build. Alan Hohn
Even when building from a stable platform, it turns out that

it's very difficult to establish an abstraction that successfully
hides the messy world of real hardware, real networks, and
real storage.
Alan Hohn, Director of Software Strategy at Lockheed Martin

@AlanHohn on DZone | @alan-hohn on LinkedIn | alanhohn.com
Alan is the author of The Book of Kubernetes (2022). He has over 25 years of professional experience in
technology, including work as a software developer, architect, lead, and manager. This has included
building complex systems fielded in some of the world's most challenging environments. He's an
experienced Agile and DevSecOps coach and currently works in Lockheed Martin Corporate Engineering.
DZONE TREND REPORT | KUBERNETES IN THE ENTERPRISE PAGE 2

ORIGINAL RESEARCH
Key Research Findings

An Analysis of Results from DZone's 2022 Kubernetes Survey
By Mike Gates, Guest Writer & Former Senior Editor at DZone
In August 2022, DZone surveyed software developers, architects, and other IT professionals in order to understand how
containers are being used and orchestrated in software development, with a particular focus on Kubernetes. We also sought to
explore the trends from data gathered in previous Trend Reports, most notably our 2021 and 2020 reports on Kubernetes.
Major research targets included:

1. The state of resource isolation, container orchestration, and infrastructure abstraction,
with a specific focus on Kubernetes
2. What Kubernetes promises vs. what it delivers
3. The kinds of applications that run on Kubernetes
Methods: We created a survey and distributed it to a global audience of software professionals. Question formats included
multiple choice, free response, and ranking. Survey links were distributed via email to an opt-in subscriber list, popups on
DZone.com, the DZone Core Slack Workspace, and various DZone social media channels. The survey was open from August
15–31, 2022 and recorded 467 full and partial responses.
Note: Like last year's survey, we recorded a large number of submissions that we considered suspect. In addition to extremely
quick response times per question (< 5s), multiple choice questions all receiving the same answer, very simple response patterns,
etc., we excluded a number of responses that were internally inconsistent — for example, those claiming to both use and not use
containers/orchestration in software projects or being both sure and unsure of using containers/orchestration. In all, we were left
with 292 responses that we considered trustworthy — with a couple of notable exceptions, which we will detail later.
In this report, we will review some of our key research findings. Many secondary findings of interest are not included here.
Research Target One: The State of Resource Isolation and Container Orchestration
Motivations (remain unchanged from our 2021 and 2020 reports):
1. Software development and runtime ecosystems are now complex and tangled enough that OS-level resource
management is often insufficient for avoiding conflicts in build and runtime environments.
2. Relatedly, given the wide array of opinions on manual SQL writing and ORM usage, we wanted to check that as more
applications run on the web, where state management is not built into the application protocol, application state management
becomes increasingly difficult to manage through explicit application-level code while being easier to automate at a lower level.
3. As software architectures increasingly take advantage of the "metal-indifference" of cloud computing while depending
on multi-platform runtimes (JVM, Node.js, and high-performance interpreted languages like Python) and complex graphs
of dependencies — often with their own rapid development cycles — a dimension for horizontal scaling that allows more
granular control over runtime environment (rather than the application source) than VM-level (needed if OS-level WORA
runtimes weren't used) becomes increasingly attractive.
4. Further, as Agile development methodologies encourage radically service-oriented designs that comprise independently
built and managed systems with well-defined system boundaries ("microservices") and strongly opaque internals,
maintenance of a single OS-level environment that serves many services' heterogeneous needs becomes increasingly
difficult — sometimes impossible.
5. Finally, as container use increases, the need for high-level container orchestration also increases.
RESOURCE ISOLATION IN LINUX SYSTEMS

Given modern container management's origins as manually configurable resource isolation in multi-user operating systems,
we've looked at those lower-level setups for the past couple of years — and we wanted to continue to do so this year.

We maintained our hypothesis that fewer respondents would report using granular, manual resource isolation techniques year
over year, although with an added caveat:
1. Higher-level containerization/orchestration tooling continues to mature, making them more attractive to

professionals and decision-makers alike.
2. Distributed architectures in which individual runtimes perform narrowly defined tasks require more automated,
source-controlled resource management.
3. DevOps professionals/SREs are absorbing more work that was previously handled by pure ops professionals (who
are more likely to use OS-level constructs).
However, at some point, we expect to see the rate of decline begin to slow down. We hypothesized last year that containers
may be reaching a saturation point in the wider software world, which would create something of a boundary to further
uptake and the choice to move away from these lower-level methods. Therefore, we repeated the question from our last two
Kubernetes Trend Reports:
What methods do you use to isolate resources in Linux systems?
Results across all responses, comparing 2020–2022 data:
Table 1
LINUX RESOURCE ISOLATION METHODS
% of Total Respondents Using
Method Used 2020 2021 2022
LXC 38.1% 32.8% 29.8%
chroot "jail" 31.2% 23.1% 27.4%
LXD 22.4% 21.6% 23.1%
LXCFS 18.8% 17.5% 18.4%
Observations:
1. At first glance, it would seem that low-level means of isolating resources are seeing a resurgence, although delving
deeper into the data reveals more to the story. When splitting out respondents who work at companies with more than
1,000 employees, the picture looks much different (see Table 2). Looking at this audience segment, a clearer story begins
to present itself:
– First, as we hypothesized, container Table 2
(and container-related tooling) adoption

LINUX RESOURCE ISOLATION METHODS AT LARGER COMPANIES
seems to be slowing dramatically.
That saturation point appears to have % of Total Respondents Using
been achieved, and it seems there is a Method Used 2020 2021 2022 (> 1,000 employees)
correlation between container adoption
LXC 38.1% 32.8% 32.9%
and company size. Note that we will
touch more on this in the next section. chroot "jail" 31.2% 23.1% 24.8%
– For smaller companies, it simply might LXD 22.4% 21.6% 21.5%

not be worth it (or within budget) to
LXCFS 18.8% 17.5% 15.4%
move into the higher-level container
ecosystem. Depending on the scope of
the projects being worked on, these lower-level abstractions might simply be a case of "If it ain't broke, don't fix it."
2. On the other hand, company size played no role in the relative rankings of each option — the order for all respondents:
LXC > chroot "jail" > LXD > LXCFS. That much matched our hypothesis last year, and the trend seems to have continued
through this year.
As we did the previous two years, we wanted to explore how these methods are being used both in development and
production environments. For the sake of consistency, we will provide the unfiltered data (Table 3) and the percentages for
companies with more than 1,000 employees (Table 4). See Tables 3 and 4 on the following page.

Results, year over year:
Table 3
LINUX RESOURCE ISOLATION METHODS BY ENVIRONMENT TYPE
Development Environments Production Environments
Method Used 2020 2021 2022 2020 2021 2022
chroot "jail" 57.6% 61.6% 66.7% 42.4% 38.4% 33.3%
LXC 53.3% 56.3% 58.1% 46.7% 43.7% 41.9%
LXD 54.9% 62.4% 50.3% 45.1% 37.6% 49.7%
LXCFS 51.8% 55.3% 50.3% 48.2% 44.7% 49.7%
Table 4
LINUX RESOURCE ISOLATION METHODS BY ENVIRONMENT TYPE AT LARGER COMPANIES
Development Environments Production Environments
Method Used 2020 2021 2022 (> 1,000 employees) 2020 2021 2022 (> 1,000 employees)
chroot "jail" 57.6% 61.6% 70.3% 42.4% 38.4% 29.7%
LXC 53.3% 56.3% 59.2% 46.7% 43.7% 40.8%
LXD 54.9% 62.4% 50.0% 45.1% 37.6% 50.0%
LXCFS 51.8% 55.3% 52.2% 48.2% 44.7% 47.8%
Observation: Last year, we hypothesized that we would see year-over-year drops in lower-level resource isolation method use
in development environments. We figured that development environments would use less complex, less scalable techniques
and, thus, would benefit from higher-level tools as they have become more user friendly. We were wrong across all counts last
year, but the waters are muddier this year. While development environments are still more likely to use lower-level isolation
methods than production, LXD and LXCFS have become more likely to be used both in dev and prod.
CONTAINER POPULARITY
Given the data, we expected to see container usage remain consistent with the numbers from the last couple of years — about
89%. Therefore, we asked:
Do you use application containers in either development or production environments? {Yes | No | I don't know}
Results since 2017:
Figure 1
PERCENT OF RESPONDENTS USING CONTAINERS: 2017-2022
UNFILTERED DATA LARGER COMPANIES

100 90.4% 88.2% 100 90.4% 88.2%
92.9%
80 70.3% 80 70.3%
82.8% % using
containers
60 60 Trendline for
44.6% 44.6%
42.2% 42.2% unfiltered data
R² = 0.7682
40 40
Trendline
for larger
20 20 companies
R² = 0.8725
0 0
2017 2018 2019 2020 2021 2022 2017 2018 2019 2020 2021 2022

Observations:
1. This is where we run into a bit of trouble with the high number of suspect responses we received. Even after we've
quarantined the obviously poor responses, it would seem that containers are now becoming much less popular over time.
That's certainly possible, but we have our doubts. However, we don't want to get into the habit of rejecting data that
doesn't fit a narrative — instead, we're going to simply provide full disclosure and walk through it. After all, as we
posited last year, this could simply be an indication that more companies, especially smaller, perhaps newer ones, are
turning to "micro-VM" serverless-supporting technologies rather than spinning up new containers.
2. Instead, we once again broke out the data by company size, and the picture became a bit clearer. Taking both datasets at
face value, it looks like smaller companies are becoming more hesitant to adopt containers in general. However, looking
at our previous surveys, there has not been a significant correlation between company size and container adoption.
To unpick this thread, we plotted the coefficient of determination (the R2 value) for both datasets. For the statistically
uninitiated, that means we're looking to see how much of the variance we see in the data can be explained by the
model (the trendline here). The closer to 1 the R2 value is, the more the model explains the variance in data. Therefore,
given that the R2 value is higher this year when looking at larger companies (0.8725) than overall (0.7682), we're going
to assume that this is a more trustworthy number in general.
3. All of that is to say, we believe container adoption remains solidly at or around the 90% mark — however, that floor might
be softer than people generally expect, and containers may not be as enticing as they once were for smaller companies.
MANAGING CONTAINERS
To create a container is one thing, but to manage them is another. And, of course, containers are a small but important part of
the wider development ecosystem. Therefore, we wanted to know exactly how people are managing those containers. With
that in mind, we asked:
What tools/platforms are you using to manage containers in development and production? Select all that apply.
Results (summed across 1,009 selections across all Table 5
categories [only displaying options that received > 5

TOOLS USED TO MANAGE CONTAINERS
votes]) are noted in Table 5.
In Development In Production
Observations:
1. In what feels like common sense rearing its head, Tool n= % n= %
it seems that most respondents use Docker Docker 108 19.5% 62 13.7%
in development, then switch to Kubernetes in
Kubernetes 82 14.8% 77 17.0%
production.
Docker Compose 74 13.3% 24 5.3%
2. Taking a walk on the cloudy side, AWS has
maintained its market dominance. The Big Three AWS EKS 51 9.2% 56 12.3%
have continued their relative rankings since time Terraform 43 7.7% 50 11.0%
immemorial (or at least 2016) — Amazon > Azure >
AWS ECS 34 6.1% 40 8.8%
Google, although GKE fared better against Azure
than we would have anecdotally expected. Ansible 34 6.1% 26 5.7%
OpenShift 28 5.0% 27 5.9%

KUBERNETES USAGE
Azure AKS 25 4.5% 26 5.7%
Now onto the titular topic — Kubernetes. Last year, we
noticed a flattening trend of Kubernetes adoption to GKE 20 3.6% 21 4.6%
match that of container adoption. With that in mind, we
Rkt 17 3.1% 13 2.9%
expected to see a number that was mostly the same —
although given that smaller companies might be moving QEMU 16 2.9% 9 2.0%
away from spinning up new containers, it's possible we'll Firecracker 8 1.4% 8 1.8%
see a slight year-over-year dip. So once again, we asked:
Does your organization run any Kubernetes clusters? {Yes | No | I don't know}
Results since 2019:
SEE FIGURE 2 ON NEXT PAGE

Figure 2
PERCENT OF RESPONDENTS USING KUBERNETES
80
78 77.0% 77.2%
76
74
73.6% 73.6%
72
70
2019 2020 2021 2022
Observations:
1. Our hypothesis was correct. Kubernetes adoption seems to have stalled and is hovering in the mid-70s percentage
range overall.
2. Once again, as we break out the data into smaller and larger companies, the same trend we've established rings true. Of
the responses, 85.5% of participants from larger companies said they ran Kubernetes clusters, whereas "only" 63.4% of
respondents from smaller companies said they did.
Historically, a chief complaint about Kubernetes has been its lack of user friendliness, and while container-related
tooling has no doubt improved over time, it's reasonable to assume that companies with fewer people and resources
just don't care to put as much effort into Kubernetes adoption as massive enterprises.
INFRASTRUCTURE ABSTRACTION
And lastly, for this research target, we wanted to examine our community's attitude toward infrastructure abstraction. We
noticed a slim but growing pessimism in last year's results, and we expected that to continue. Just as we asked in 2020 and
2021, we asked this year:
Please select the option that best describes your attitude toward infrastructure abstraction in 2022.
Results, year over year:
Table 6
ATTITUDES TOWARD THE CURRENT STATE OF INFRASTRUCTURE ABSTRACTION
% of Total Respondents
Opinion 2020 2021 2022
Infrastructure abstraction in 2022 is excessive and getting out of hand 16.0% 18.4% 21.2%
We're finally getting close to pure, non-leaky infrastructure abstraction 21.3% 18.8% 26.3%
The cost in complexity of modern infrastructure abstraction is worth the

47.9% 46.2% 44.9%
benefits of infinite scaling and continuous delivery
No opinion 13.7% 15.5% 7.6%
Observations:
1. The results here are telling — respondents are feeling worse about infrastructure abstraction. The percentages grew
slowly but steadily in the attitudes of "getting out of hand" (18.4% in 2021 vs. 21.2% in 2022) and "not worth the cost" (the
inverse of the 46.2% in 2021 vs. 44.9% in 2022).

2. However, they are also feeling much better about the state and near future of infrastructure abstraction (18.8% in 2021 vs.
26.3% in 2022) thanks to a (relatively) giant shift in the "no opinion" camp. Roughly half of previous years' undecided voters
made a choice, and while some of them seem to be getting fed up with modern infrastructure abstraction, most of those
respondents are pleased with what they see.
3. It's worth repeating that the biggest camp is still those who think the gains are worth the complexity, which is unsurprising.
Meanwhile, it will be interesting to see whether that newfound optimism among the "pure, unleaky" camp sours over the
coming year, or if advances and maturation within the field will stem the flow of the more pessimistic parties.
Research Target Two: Kubernetes — Promise vs. Performance

Motivations:
1. It's easy to forget that only a handful of years ago, Kubernetes was an experimental, bleeding edge technology that few
had implemented, let alone mastered. Of course, now that adoption is the norm and Kubernetes has had time to mature,
it's worth seeing how Kubernetes has been tweaked and tuned to simplify difficult tasks.
2. On the other hand, now that Kubernetes is not quite as "buzzwordy," we wanted to keep tabs on what it might not
have been up to snuff for. After all, the list of anecdotes about companies implementing the hot new tool or technique,
regardless of how optimal it is for a given use case, would be longer than this Trend Report.
3. And finally, just like in 2021, we wanted to gauge the community's feelings on Kubernetes infrastructure and workload
state matching, considering the importance of solving that problem for distributed systems.
KUBERNETES: WHAT IS IT GOOD FOR?

First, we'll start off looking at the positives. While it's easy to hate on what's popular, popular things are often popular for a
reason — and Kubernetes is no exception. As time passes, we expect those improvements to grow over time.
Considering that, we're going to take another look at a question we asked in both 2020 and 2021:
What has Kubernetes improved at your organization?

Table 7
Results, year over year are shown in Table 7.
IMPROVEMENTS FROM USING KUBERNETES
Observations:
1. A few specific trends immediately jump out % of Total Respondents
at us. While Kubernetes was, unsurprisingly, 2020 2021 2022
Area
seen as helpful for CI/CD, that trend increased
Deployment in general 66.0% 64.3% 61.7%
significantly this year. This is probably a sign
of growing competence and confidence with Autoscaling 65.0% 62.8% 63.8%
Kubernetes for one of its primary functions.
CI/CD 63.7% 61.1% 73.8%
We also saw a big jump in people's thoughts
about Kubernetes' effect on modularity after it Building microservices 53.6% 52.3% 47.5%
had been previously stable. Reliability 46.0% 39.6% 48.2%

As modularity becomes more widely
Application modularity 44.3% 43.3% 51.1%
implemented, it's possible, even likely, that
Architectural refactoring 36.1% 32.2% 31.2%
Kubernetes tooling and experience has
morphed to support it. Overall system design 33.5% 32.2% 32.6%
2. We were suspect of last year's drop in Cost 28.9% 28.2% 34.8%
reliability, and that caution served us well. Security 24.9% 31.3% 31.2%
That dip was erased — and then some — this
Other 3.2% 2.6% 4.3%
year as nearly half of respondents considered
Kubernetes to help with reliability. This could
be a sign of differing audience demographics year over year or simply a sign that Kubernetes was on the wrong track and
course corrected in a pretty spectacular fashion.
3. Last year's jump in security has held firm. It's still not a popular opinion, but Kubernetes' impact on security seems to have
stabilized year over year.

4. Interestingly, there appears to be a closing of the gap with costs. While we haven't published the results here in tabular
form, we also asked the question, "What has Kubernetes made worse at your organization?" Cost was far and away the
biggest pick (37.0%). It will be interesting to see whether this year's increase holds steady or even overtakes the opposing
point of view — or, instead, declines back down to 2020's level.
For the curious, the next closest picks for what Kubernetes made worse were security (26.1%) and architectural
refactoring (24.4%). Every other area was selected at rates in the teens.
5. Otherwise, the story being told seems to be one of stabilization and mild decline. For instance, fewer than half of
respondents now think Kubernetes makes building microservices better as opposed to last year (52.3%) and 2020 (53.6%),
architectural refactoring dipped slightly, and autoscaling rose a bit after last year's minor dip.
THE STATE OF STATEFULNESS

Moving on from broad categories to something more specific, we wanted to re-examine how our community feels about
Kubernetes' ability to handle stateful workloads. Given the choice, there are other options out there to handle state
maintenance that professionals are probably much more familiar with. However, as people grow more comfortable with
Kubernetes, and as Kubernetes matures, we wanted to keep an eye on this trend. So we asked:
Do any of your organization's Kubernetes clusters run any stateful workloads where state is maintained within the cluster?
Results, comparing 2020–2022 data:
Table 8
RUNNING STATEFUL WORKLOADS WHERE STATE IS MAINTAINED WITHIN A CLUSTER
2020 2021 2022
Yes 45.7% 49.6% 52.1%
No 40.1% 33.8% 33.3%
I don't know 14.2% 16.7% 14.6%
Observation: The trendline here is simple to read — Kubernetes has continued its gradual incline in popularity of maintaining
state while running stateful workloads.
However, we also wanted to double check how happy people are with that arrangement, so we asked:
Are you glad that your Kubernetes cluster(s) maintain(s) state within the cluster(s)?
Results, comparing 2020–2022 data:
Table 9
APPROVAL OF KUBERNETES' STATEFUL WORKLOAD MANAGEMENT
2020 2021 2022
Yes, Kubernetes is doing a good job of

73.8% 77.0% 75.5%
maintaining state
No, I wish we were maintaining state

17.3% 19.5% 12.2%
outside the Kubernetes cluster
I have no opinion 8.9% 3.5% 12.2%
Observations:
1. Overall, people are still happy with the job Kubernetes is doing in handling this thorny problem. Three in four respondents
is a pretty decent line to maintain all things considered.

2. More interesting is the (relatively) massive drop in the number of respondents who were previously dissatisfied with
Kubernetes' ability to maintain state within a cluster. In both cases, the dips seem to have gone straight to the "no
opinion" camp. On the whole, it appears that Kubernetes has done a good job of at least making people reconsider it for
maintaining state, even if it hasn't completely sold previous naysayers on its capabilities.
Lastly, for this section, we wanted to take a quick look at how exactly our community was maintaining state with Kubernetes.
Therefore, we asked:
What construct(s) does your organization use to maintain state within a Kubernetes cluster?
Results:
Figure 3
HOW STATE IS MAINTAINED IN KUBERNETES CLUSTERS
1.1%
16.5%
StatefulSet
DaemonSet
22.0% 60.4% CustomResourceDefinition
Other - write in
Observations:
1. Unsurprisingly, the classic StatefulSet is seen as the go-to option for most users, although it will be interesting to see how
this result changes over time.
2. For example, as the importance, prevalence, and number of projects involving monitoring and logging continue to grow,
we will see how the DaemonSet camp adjusts with respect to the other options. For what it's worth, smaller companies
seem somewhat more likely to rely on StatefulSets vs. DaemonSets than larger ones, which could be an indication of
those monitoring and logging functions of big projects.
Results:
Figure 4
MAINTAINING STATE IN KUBERNETES CLUSTERS BY COMPANY SIZE
70
60
50
> 1,000
40 employees
30 < 1,000
employees
20
10
0
StatefulSet DaemonSet CustomResourceDefinition Other - write in

Research Target Three: What Runs (Well) on Kubernetes Clusters?
Motivations:
1. Moving one step beyond cluster management itself, we wanted to examine the applications and workloads handled
specifically with help from Kubernetes.
2. Given the importance of scalability in the modern development paradigm, we wanted to see what kinds of autoscalers
our community is choosing when working with Kubernetes and look at what those results indicate about ideal use cases.
TYPES OF WORKLOADS THAT RUN ON KUBERNETES CLUSTERS

The old adage, "Use the right tool for the right job," is as true now as it ever was. Obviously, using Kubernetes for some tasks will
likely be better/more efficient than others. As with last year, we wanted to see if using Kubernetes for some workloads resulted
in more satisfaction with it than others.
So we asked:
What types of workloads does your organization run on Kubernetes clusters? Select all that apply. {Web applications | CPU-
intensive | GPU-intensive | Memory-intensive | Storage/database-intensive | Non-web general compute | Other}
Results:
Figure 5
TYPES OF WORKLOADS RUNNING ON KUBERNETES CLUSTERS
100
80
60
40
20
0
Web apps CPU-intensive GPU-intensive Memory- Storage/database Non-web Other
intensive -intensive general compute
Observations:
1. In a change from last year, we broke out non-web general compute into its own category for 2022. Last year, we wanted
to see how Kubernetes handled comparatively ephemeral connections with strong connection-level transaction
semantics, so we simply combined both options. Last year's number for those combined figures was 84.6%, but we could
not confirm whether web apps were especially popular choices for Kubernetes clusters. This year, however, we can safely
conclude that yes, web applications seem to be the primary use case for Kubernetes.
2. We also wanted to follow up regarding the least popular option: GPU-intensive workloads. Kubernetes's ability to
schedule GPUs is still in beta, but we saw a year-over-year increase from 15.4% in 2021 to 17.2% this year. Last year, we were
impressed that an experimental feature saw even 15%, so it's unsurprising that given another year of more or less the
status quo, there's been a slight uptick in usage.
3. It's also worth noting that we didn't see quite so obvious a split between the categories of this question and which
respondents thought infrastructure abstraction was getting out of hand as we did last year. In 2021, we noted that respondents
whose organizations worked with GPU-intensive workloads were most likely to think that infrastructure abstraction was
getting out of hand, whereas those working with storage/database-intensive workloads were least likely to feel that way.
The "GPU-intensive" camp still feels that way, but it seems the "storage/database-intensive" crowd has begun to
feel the same. However, respondents whose companies worked with memory-intensive workloads were most likely

to think that "we are finally getting close to pure, non-leaky infrastructure abstraction," and the non-web/general
compute respondents were most likely to think that "the cost in complexity of modern infrastructure abstraction is
worth the benefits of infinite scaling and continuous delivery." We'll keep an eye on those figures to see if it might be a
one-year reversal, as we've seen with other stats in this report.
TRAFFIC PATTERNS AND KUBERNETES

Scalability is one of Kubernetes' chief benefits. The ability to handle highly variable traffic without overcommitting resources is
one of those benefits that only gets better as your traffic grows. Therefore, we asked:
What kinds of usage/traffic patterns do your organization's Kubernetes clusters serve? {Predictable and frequent (short,
known idle periods between work) | Predictable and infrequent (long, known idle periods between work) | Unpredictable with
low variance (unknown heavy-usage times within narrow range) | Unpredictable with high variance (unknown heavy-usage
times not within narrow range) | A mix of all the above}
Results:
Figure 6
TYPES OF USAGE/TRAFFIC PATTERNS SERVED BY KUBERNETES
50
40
30
20
10
0
Predictable/ Predictable/ Unpredictable/ Unpredictable/ Mix
frequent infrequent low variance high variance
Observations:
1. This data is similar to last year's, with a couple of notable exceptions. First, more respondents chose a mix of all the above
over predictable, frequent workloads, which seems to emphasize that most professionals see Kubernetes as a strong
general-purpose option for orchestration. That being said, it's clear that lightweight services (signified by the short,
known idle periods between work) are still considered desirable when working with Kubernetes.
2. Second, unpredictable workloads with high variance overtook those with low variance, although they were still close
in 2021. It's worth pointing out that both percentages went down, but the low-variance figure dropped much more
heavily. This could be one of those one-year reversals, or it could be a sign that there's just a better option when there are
unpredictable bouts of heavy amounts of work within a narrow range.
AUTOSCALERS
And speaking of scalability, we wanted to end these research findings by examining which autoscalers are preferred in
Kubernetes clusters. We asked:
What autoscalers does your organization use in its Kubernetes clusters? Select all that apply
Results:
SEE FIGURE 7 ON NEXT PAGE

Figure 7
AUTOSCALERS USED IN KUBERNETES CLUSTERS
80
60
40
20
0
Vertical Horizontal Cluster Other - write in
Observation: Horizontal autoscaling actually increased its gargantuan lead over last year, where it hovered around 65%. We're
going to parrot last year's sentiments that because horizontal autoscaling is (generally) more opinionated, Kubernetes is being
used mostly to solve easier problems when it comes to cluster management.
Future Research
These key research findings were an attempt to understand and explain the trends of container adoption in general and
Kubernetes in particular. Given the relative saturation of containers in modern development, it is perhaps expected that we are
seeing rates stabilize. However, we'll also keep an eye on that potentially soft ~90% mark when it comes to container adoption,
especially when considering company size.
We will also continue to monitor data on workload types, traffic/usage, and autoscaler choice to see whether those figures
change in the coming year. And, of course, we want to monitor the pros and cons of Kubernetes as this powerful tool continues
to grow and mature in the ecosystem.
Please contact publications@dzone.com if you would like to discuss any of our findings or supplementary data.
Mike Gates, Guest Writer & Former Senior Editor at DZone

@Michael_Gates on DZone
Mike has spent more than five years working with DZone contributors and clients alike, with roles ranging
from frontline web editor to editorial consultant to managing editor. These days, he uses his history of
recognizing readership trends to write and edit articles for DZone.com — when he isn’t busy listening to
audiobooks and searching for the perfect oven-baked beef brisket recipe.

Kubernetes Done Right.
Boost EKS
with D2iQ.
Take command
with a unified view.
Simplify Multi-Cluster Multi-Tenant

Management with AWS EKS and D2iQ.
Get the Cheatsheet

PARTNER CASE STUDY
Case Study: Cerved

A Trusted Partner Makes a Difference: Kubernetes Adoption With D2iQ
Challenge
Based in Italy, Cerved is the premier provider of credit risk analysis and other
business information, and has been in business since 1974. In modernizing
its IT infrastructure, Cerved aimed to gain improved business agility and the
benefits of a containerized environment. However, moving to a container and COMPANY
microservices platform required not only a transformation in architecture, but Cerved
also required Cerved's IT staff to acquire new Kubernetes management and
developer skills. COMPANY SIZE
5,000 employees
Solution
INDUSTRY
Moving Cerved's entire business to Kubernetes in the cloud was a major
Information services
undertaking that involved risks such as disruptions and downtime.
PRODUCTS USED
Cerved made the migration to cloud-native Kubernetes in steps, first moving
Kubernetes, D2iQ Kubernetes Platform
its development and test environments to D2iQ Kubernetes Platform (DKP)
(DKP), Amazon Web Services (AWS)
running on Amazon Web Services (AWS). After this successful migration,
Cerved successfully moved its production environment to DKP on AWS.
PRIMARY OUTCOME
Cerved's main goals were to accelerate
Results
time to market, employ a DevOps
The goal was to be more agile and faster to market — from marketing to the
approach, and gain more agility, stability,
production of services — and to scale more easily and gain more stability. The
and scalability.
DKP platform delivered on all of these promises.
On the DevOps side, production issues are easier to handle. In the case of a
failure, it's quick and easy to patch or roll back. The openness and portability "D2iQ's platform offers simplicity. The
of DKP is another major benefit, making it easy to move to Google, Azure, or fact that it is basically a pure Kubernetes
any other cloud provider. It's also easy to reproduce services on another site, open-source platform means there is
on-premises, or in the cloud. flexibility."
Going forward, Cerved plans to expand the scope of the environment to more — Damiano Tarantino,
clusters and to shorten the AI lifecycle for its data scientists, making it easier IT Architecture &
to move from idea to models to production. Infrastructure Director, Cerved
Overall, the benefits of the DKP deployment included:
• Agile development CREATED IN PARTNERSHIP WITH
• Platform stability and scalability

• Faster time to market
• Openness and portability derived from DKP's pure open-source
Kubernetes
• Flexibility to extend capabilities, including AI

CONTRIBUTOR INSIGHTS
Kubernetes: Beyond Container

Orchestration
Serverless Reimagined With Kubernetes
By Pratik Prakash, Master Software Engineer (SDE-IV) at CapitalOne
Kubernetes today is the most prominent and prevailing container orchestration engine. Kubernetes became central to
cloud-native computing because it is open source and has a rapidly growing ecosystem. If we observe its evolution and
adoption trend in last few years, especially around the cloud native world, it is more than "just" a container orchestration tool.
It has outgrown being a container orchestration engine and is now a frontline building block of a next-generation cloud-
native ecosystem.
Developers are experimenting with Kubernetes by reimagining and reestablishing it as much more than just a container
manager. We will discuss a few of its established enterprise patterns and buzzing use cases in the open-source community.
Kubernetes as a Server of Serverless Computing

Enterprises with exposure and experience around Kubernetes have identified the full potential of Kubernetes as the platform
that is the "server of the serverless." Serverless is now reimagined with Kubernetes, and Knative (kn) was recently launched by
CNCF (the parent of Kubernetes) to abstract Kubernetes as Serverless computing. At its core, Knative consists of three modules:
1. Build – build images from source
2. Serving – deploys functions (image build) on Kubernetes clusters and maps scaling, routing, etc.
3. Eventing – maps events and messaging ingestion with servicing
Like kubectl for Kubernetes, kn is the new, hot command on Figure 1: Serverless on top of Kubernetes
the terminal for enabling Function as a Service (FaaS) on top
of Kubernetes. It takes a container image as a building block
and handles everything on top of Kubernetes. Kubernetes is
serverless for containers and is similar to what AWS Lambda is
to function. Kubernetes as a serverless platform is accelerating
its adoption across the enterprise; however, its peers like
AWS Lambda, Cloud Functions, and Azure Functions are still
dependent on vendor lock-in (i.e., they only work if you deploy
on their cloud).
Unlike its peers, Kubernetes addresses some of the existing

challenges like constraint in size of the artifact, compliances
and regulations, data sovereignty, and granular control within
an enterprise. The main architecture differences between
serverless Knative on Kubernetes versus conventional
Kubernetes come from an additional layer of abstraction.
Kubernetes as Serverless eliminates repetitive configuration
and build tasks.
Knative is a new addition to the Kubernetes ecosystem, and it

holds the potential to disrupt on-premises serverless options
to build event-based architecture. Features like "scale to zero,"
"scale from zero," and in-cluster build makes Kubernetes a Overview of Kubernetes cluster with Knative as a serverless and
complete serverless platform. event-driven platform

KUBERNETES AS A BIG DATA AND MACHINE LEARNING PLATFORM
Kubernetes is being seen and widely adopted as a platform for big data processing and stateful data services among data
science and machine learning (ML) tech stacks. It abstracts underlying infrastructure and optimizes the provisioning of elastic
compute, which combines GPU and CPU under the hood. Kubernetes is great for machine learning as it natively has all the
scheduling and scalability that ML requires.
Containers and Kubernetes combined are powerful and flexible for building big data software as compared to conventional
data cluster setups, which have the complexity of distributed cluster management and compute scale overhead. Kubernetes
leverages on-demand GPU and CPU computing to enhance big data and ML processing. Kubernetes can deliver GPU-
accelerated computing and network solutions for running ML and NLP processing on the edge. Kubernetes running on a
co-processor is becoming an important part of future computing. Its dynamic resource utilization is beneficial for data science
workloads, while training models or feature engineering demands can scale up and down very quickly.
Figure 2: Kubernetes-powered big data and ML cluster
On-demand scalable coprocessor for ML processing
Frameworks like KubeFlow, Spark, Hadoop, PyTorch, TensorFlow, and more are now adopting containers. Stateful MLOps
are widely adopting containers and Kubernetes for spinning multiple clusters, processing large training and test sets, and
storing learned models. To simplify handling data modeling frameworks, one option is Kubeflow, an open-source ML toolkit for
Kubernetes to run declarative, configurable jobs to execute.
Kubeflow abstracts and orchestrates complicated big data and machine learning pipelines that are running on top of
Kubernetes. Kubernetes under the hood of ML pipelines is a staple for MLOps. It makes it easy for data scientists and machine
learning engineers to leverage hybrid cloud (public or on-premises) for processing elasticity and scalability. Kubernetes enables
building a neutral big data platform to avoid vendor lock-ins of cloud-managed services.
KUBERNETES AS A FEDERATION FOR HYBRID AND MULTI-CLOUD CLUSTER FOR THE ENTERPRISE
Kubernetes is infrastructure agnostic and is leveraged by enterprises for sophisticated container-based cluster federation.
Kubernetes helps to combine a hybrid or multi-cloud setup into a single platform lump for obvious benefits.
Managed services are not the solution for everything. There is always a struggle between balancing the convenience of a public
cloud versus a private sovereign cloud. Kubernetes is seen as an answer for this, as it enables multi-cloud access for seamless
application delivery across the industry standard API (Kubernetes interface). It helps to achieve regulatory compliance for
enterprises with Kubernetes by abstracting private and public cloud integration as single federated platform.
Enterprises leverage Kubernetes to avoid vendor lock-in by providing flexibility around hybrid and multi-cloud clusters. It's
interesting to observe how current cloud-based architecture patterns have adopted Kubernetes. Cloud-based enterprises along
with the open-source community have realized that Kubernetes is not just a container management tool. It is now evident that
Kubernetes is a complete platform to host lifecycles for running an application on a hybrid or multi-cloud paradigm.

Kubernetes has been widely adopted as Platform as a Service (PaaS). However, during early days, Kubernetes official
documentation mentioned it as Container as a Service (CaaS). Recently, it is observed that Kubernetes adoption patterns and
usage are way beyond CaaS, hence the updated documentation of Kubernetes as next-generation PaaS.
Kubernetes is new age PaaS, meaning:
• Kubernetes in conjunction with hyperconverged infrastructure (HCI) is a new private or hybrid cloud alternative. It gives
enterprises full control of their services and regulatory sanctity.
• Kubernetes enables enterprises to achieve a single abstracted and simplified* platform to operate SRE, DevOps, and
DevSecOps on top of a hybrid and multi-cloud span. (*That's debatable for sure.)
Figure 3: Kubernetes as new generation PaaS
Hybrid and multi-cloud containers federations as new generation PaaS
Conclusion
Generally, large organizations are not comfortable giving up platform controls to AWS Lambda, AWS Fargate, or Azure
Functions. Kubernetes has become a de facto choice that provides the best of both worlds — control and on-prem sanity
along with cloud-native compute elasticity, scalability, and resiliency that comes from a declarative containerized ecosystem.
Kubernetes with open-source tools like Helm (IaC), Grafana (telemetry dashboard and alerting), and Prometheus (metric
ingester) is making it a perfect recipe for a DevOps-enabled and SRE-friendly ecosystem for the enterprise.
Data science and machine learning enterprises are accelerating Kubernetes adoption as a big data processing platform. Lately,
ML and big data frameworks are becoming containerized, which makes Kubernetes a preferred choice for a data clustering
and modeling ecosystem under the hood. This preference also includes Kubernetes' abstracting elastic GPUs and CPUs plus its
on-demand scalability with statefulness.
Kubernetes abstraction frameworks like Knative are taking Kubernetes to another level. Kubernetes is becoming a new
server for serverless architecture, and it is gaining momentum. It is providing open-source alternatives for AWS Fargate and
OpenShift. Kubernetes is "the wolf in sheep's cloth"; it has been reimagined into many shapes and forms beyond being "just" a
container orchestration tool.
Pratik Prakash, Master Software Engineer (SDE-IV) at CapitalOne

@scanpratik on DZone | @pratik-prakash on LinkedIn | @PratikPrakash_ on Twitter | @scanpratik on GitHub
Pratik is an experienced solution architect and a passionate engineer with hands-on multi-cloud and
data science expertise. He is an open source advocate and enjoys community participation around Java,
Kotlin, and the cloud native world. He has worked with companies like UnitedHealth Group as a senior
staff engineer and with Fidelity International as an expert engineer where he led engineering teams for cloud-native
application modernization.

The Importance of Kubernetes

in the Cloud-Native Ecosystem
By Pavan Belagatti, Developer Advocate at Harness
If you are a developer or an engineer working in the software industry, then you have undoubtedly noticed the complexity of
managing resources across different services and applications. This is especially true when your company has scaled beyond a
single data center or environment. With so many stakeholders involved in various projects, it becomes difficult for everyone to
keep track of where everything stands at any given time. This is where Kubernetes comes into the picture.
There are several reasons why Kubernetes is gaining popularity among modern businesses:
• Kubernetes can improve productivity and scalability, reduce the risk of downtime, and enable continuous delivery
of applications.
• Kubernetes comes with a rich set of features out of the box, making it easy for you to set up and manage a containerized
application lifecycle from development to production.
• Kubernetes is open source, and its development has been driven by the community. It is also highly extensible, allowing
third-party vendors to build tools and services that can integrate with Kubernetes and make it even more powerful.
• Kubernetes supports all major operating systems and is also compatible with most standard programming languages.
Kubernetes is a powerful tool for accelerating your journey to the cloud-native world, making it easier for you to manage
containerized microservices and accelerate your adoption of DevOps practices. In this article, we will see why Kubernetes is a
must-adopt cloud-native tool for businesses and how it helps them speed up their software deployment.
Kubernetes: King of the Cloud-Native Ecosystem

Today, enterprises of all sizes are adopting cloud-native computing to accelerate their digital transformation. Kubernetes
continues to be the leading provider in the cloud-native ecosystem and is becoming an essential tool for anyone looking to
adopt a cloud-native strategy.
Figure 1: Kubernetes architecture
Image source: Nived Velayudhan, CC BY-SA 4.0

Kubernetes can help you achieve a cloud-native ecosystem by automating the process of setting up an environment where
you can host your containerized applications. This can be done with just a few simple commands. Kubernetes can also help you
deal with the complexity involved in managing your application resources across different services and applications that are
distributed across multiple environments.
Kubernetes handles the entire lifecycle of your application containers, including the orchestration, configuration, and
scheduling of containers across a set of machines. In addition, Kubernetes is designed to be highly scalable and fault-tolerant,
making it ideal for continuous delivery of your code and enabling you to achieve continuous delivery with less manual effort
and better collaboration between your teams.
The most significant benefit of Kubernetes for DevOps and the cloud-native ecosystem is that it alleviates burdens. It allows
engineers to share their dependencies with IT operations. It also resolves conflicts among different environments. It allows
the engineers to handle customer needs while relying on the cloud for many functioning applications. Kubernetes simplifies
container tasks, such as canary deployment, rolling updates, and horizontal, vertical, and dynamic auto-scaling. It is critical to
the DevOps managed services to streamline development, testing, and deployment pipelines.
Kubernetes can assist with continuous application delivery by rolling out updates in a consistent and streamlined fashion. The
application can be orchestrated using containers. In order to deploy multiple applications using Kubernetes, each one of them
is placed in a distinct container. Since the updates can be performed in one part of the application by restarting the container
without impacting the remainder of the software, they can be accomplished in one part of the app.
LIFE BEFORE KUBERNETES

It all started with big monolithic applications, which involved the complexities of managing and delivering software. It used
to take months for developers to push their changes to production. There used to be severe downtimes, and businesses used
to incur losses due to insufficient uptimes. It was a headache for organizations to make sure their services handled ongoing
changes smoothly without affecting any services. That is when microservices came along as a boon, where a humongous
monolithic application breaks down as several services, and each service is handled by a group of developers.
Each microservice behaves as an individual service without affecting the other service. This helped developers as well as
organizations to deploy software in chunks with increased frequency. During this time, Docker was introduced to deploy these
microservices with the containerization concept.
Figure 2 below shows a simple representation of microservices architecture with various microservices involved in an
e-commerce website:
Figure 2: Representation of microservices
Docker completely revolutionized software packaging and delivery with containers, helping organizations to deliver software
with speed. While containerization made a big move, there was a need for an orchestrator to manage these containers, and
that is how Kubernetes was born.
Consider a scenario where a start-up is making use of Docker containers. As the startup grows, the number of applications
also increase and, similarly, so do the containers. It is easy to manage these containers when their number is lesser, but as the
number starts to grow, it becomes difficult and you need an orchestrator to manage these containers. This is exactly where
Kubernetes shines and helps you manage the containers with its unique features of scalability, security, self-healing, high
availability, and much more.

Figure 3: Application delivery flow
LIFE AFTER KUBERNETES

Google introduced Kubernetes in mid-2014, and later Microsoft, Red Hat, IBM, and Docker joined the community. Kubernetes
made every developer's and SRE’s life easier by helping them coordinate well and manage the workloads efficiently. When
used efficiently, Kubernetes:
• Removes downtimes
• Helps in application scalability according to the traffic pattern
• Helps in securing the application and the Secrets by incorporating security policies
• Allows running a plethora of applications in production
• Allows applications to roll back if something goes wrong
• Increases developer productivity by automating the tasks associated with containers.
With Kubernetes, you can deploy new projects more quickly. A reusable pipeline and new loads as code modules can now
be distributed across a variety of projects. It can be difficult for development teams to keep tabs and monitor infrastructure
operations if they are not automated. Unexpected traffic spikes or power outages, for example, might occur. When the
application goes down, it is not a good sign for the organization. With Kubernetes, you can automate scaling and update
patches to address such issues.
Let’s consider an example to help understand the importance of Kubernetes in the cloud-native ecosystem. Let’s say you have
a production app that handles customer orders. You don’t want to push new code to that app unless it’s been thoroughly
tested and meets certain criteria. Traditionally, you would wait until your testing team has confirmed that the new code can be
deployed safely, and then you would wait for an operations engineer to perform the deployment manually. While this model
works, it’s incredibly time-consuming. This is especially true if you have multiple apps that require this type of attention.
If your app fails, you might need to wait several days before retrying the deployment. This causes your organization to lose out
on valuable production time while you wait for the system to be ready again. With Kubernetes, you can use continuous delivery
to deploy new code to your production app every time someone pushes a new version to the code repository. This ensures that
nothing is deployed to production unless it meets your organization’s deployment criteria. Kubernetes also has a built-in self-
healing mechanism that can quickly detect when nodes go offline and bring them back online.
KUBERNETES USAGE: NOTABLE CASE STUDIES

Now, Kubernetes is being used by all the big organizations that like to deploy their software fast with scalability. Kubernetes has
become a de facto container orchestration tool in the cloud-native world. Let's look at some notable examples of Kubernetes
usage in different organizations:
• The German automaker Mercedes-Benz makes use of 900 Kubernetes clusters to support its wide range of products
across the globe.
• PayPal recently started scaling Kubernetes to over 4,000 nodes and 200,000 Pods.
• HBO found that in times of peak demand for Game of Thrones, it was running out of available IP addresses to help deliver
the content to viewers, and HBO chose Kubernetes to solve this problem along with scalability issues. "We went from not
running a single service inside of a container to hosting all of the Games of Thrones Season Seven with Kubernetes," Illya
Chekrygin, Senior Staff Engineer at HBO, told the 2017 KubeCon audience.
• Kubernetes helped Tinder Engineering drive innovations toward containerization and low-touch operation through
immutable deployment. Application build, deployment, and infrastructure would be defined as code. They were also

looking to address the challenges of scale and stability, and they solved interesting challenges by migrating 200 services
and running a Kubernetes cluster at scale.
• Pinterest used Kubernetes to scale its workloads, and by the end of 2020, they orchestrated 35,000+ Pods with 2,500+
nodes in their Kubernetes clusters.
Conclusion
From its origin as an open-source project within Google to becoming a leading force in the enterprise cloud-native space,
Kubernetes has come a long way. Today, enterprises and developers from all over are finding new ways to leverage Kubernetes
for operations and application development. Kubernetes has become the standard approach for deploying and managing
containerized applications. The Google-backed project has grown significantly in adoption since its inception. If you’re looking
to accelerate your organization’s digital transformation journey, you should consider using Kubernetes. While it’s not a tool
that every organization will need, it is an essential piece of any modern IT toolkit. Kubernetes has gained a special place in the
cloud-native ecosystem, and in fact, it has become a synonym for the word "cloud-native."
Pavan Belagatti, Developer Advocate at Harness

@PavanShippable on DZone | @pavan-belagatti-growthmarketer on LinkedIn | @Pavan_Belagatti on Twitter
Pavan is a global DevOps influencer and technology writer who loves talking about Kubernetes and
experimenting with Node.js applications. As a developer advocate at Harness, he wears many hats,
including writing articles, talking at meetups and conferences, and helping developers learn how to
deploy software efficiently. In his free time, he shares DevOps stories on LinkedIn, sings Bollywood songs, and plays cricket.

Advancements in Cloud-Native
and Kubernetes Observability
By Marija Naumovska, Co-Founder & Technical Writer at Microtica
In today's world, it's more important than ever to have visibility into your system's performance and health. Modern applications
rely on complex microservices architectures and cloud-native technologies, like Kubernetes. Observability helps us understand
not just application behavior, but also infrastructure configuration changes and dependencies, as they happen in real time.
What Is Cloud-Native Observability?

Cloud-native observability is the ability to understand the health and status of a system based on the external data exposed by
its elements, like containers, microservices, Kubernetes, and serverless functions. Cloud-native observability is built on three
postulates: monitoring, logging, and traceability. By understanding these postulates and their implications, you can begin to
understand why observability is essential in modern DevOps practices. 
Figure 1: The three pillars of observability 
Monitoring, or metrics, measure the health and performance of applications and their infrastructure — these are quantifiable
measurements. Metrics provide real-time alerts of resource statuses and are critical for collecting insights into how fast your
application responds and/or detects early indicators of performance problems.
Another pillar of observability is logging. Logging captures detailed error messages as well as stack traces. Logs are records of
events, warnings, and faults that occur inside a system. They are a great source of visibility as they contain information like the
time an event took place and who or what endpoint was associated with the event.
By combining logs and traces, you can get the full context of a system's availability. Tracing helps to investigate an issue or
performance bottleneck in a containerized or microservices-based ecosystem by collecting data of the entire journey of an
application request that moves through all the layers and nodes of that system.
WHY IS OBSERVABILITY IMPORTANT? 

Observability is very important for developers and DevOps engineers because it provides the necessary information needed
to track an entire system's performance and health, troubleshoot and diagnose issues quickly, and make informed decisions
about the next steps needed to fix a problem.

Observability in cloud-native environments, such as Kubernetes, can help to understand what's going on within the clusters,
so you can optimize compute power without compromising performance. Another great use case where observability helps is
cloud cost management. To avoid unnecessary compute costs, you need to monitor clusters and understand your application's
resource usage and needs. 
Security issues and vulnerabilities can be detected quickly with good monitoring and observability tools in place. 
Common Challenges in Observability and Monitoring

Kubernetes is one of the most popular open-source container runtime engines because of its versatility, reliability, and ability
to provide abstraction among cloud providers. However, monitoring Kubernetes can be a quite challenging task for teams that
require a diverse set of monitoring tools and solutions. 
HIGH QUANTITY OF COMPONENTS AND METRICS

Working with Kubernetes brings its own set of challenges; it has more components than traditional infrastructures, like
clusters, nodes, Pods, namespaces, Services, and more. These components produce their own metrics, which can be really
helpful but also overwhelming if you don't know how to understand them. 
THE COMPLEXITY OF MICROSERVICES

Modern applications can include numerous microservices, and Kubernetes needs to handle their availability and scalability by
keeping track of all of them. Each Service can be distributed across multiple instances, forcing containers to move across your
infrastructure as needed. Additionally, microservices need to be in constant communication with each other, which is also
done inside the Kubernetes cluster. 
Figure 2: Layers of Kubernetes infrastructure

DYNAMIC CONTAINERS 
Depending on the organization, the container's life can be very
short (from 10 seconds to 5 minutes). This creates something
known as "pod churn" — the lifecycle through which new
containers and Pods are created, used, destroyed, and recreated.
Kubernetes manages these elements to make sure that there

are available resources at any time and that they are allocated
where needed.
With every new deployment, Kubernetes decides to move,

recreate, or destroy Pods as necessary. If there is a downscaling
need for the application, Pods or nodes can disappear forever.
This means that once a container dies, all of the information
inside is gone. 
Advancements in Kubernetes Observability

Now that Kubernetes is more popular than ever, observability
tools have also become more sophisticated. Some of the key
advancements in Kubernetes observability include real-time
monitoring, performance analytics, and application visibility.
EMERGENCE OF OBSERVABILITY TOOLS

In the traditional development lifecycle, monitoring and observability haven't been exactly part of the whole process. But,
now as cloud-native systems grow, and infrastructures become more complex, observability is becoming the focus for
organizations. The need to maintain system stability, reliability, and performance is what enables the growth and maturity of
observability and analysis tools.
In the past few years, cloud-native observability tools have made huge advancements. Many different tools have emerged for
metrics monitoring, collecting data, and analyzing logs and traces from individual parts of your cluster.
ACHIEVING CLUSTER VISIBILITY

The need to aggregate and relate all of this observability data from different sources and get a holistic view of your entire
system is much bigger now. This is why different open-source tools can be easily integrated with each other, to create complete

visibility for developers. Cluster visibility tools can aggregate metrics from different microservices and applications in a data
center, give insights into performance during deployments, understand how active services are behaving across the data
center, pinpoint application-level issues, and provide real-time alerts to administrators.
To achieve this goal, most of these tools offer native integration with monitoring systems so that you can get notified if any
service goes down unexpectedly or is experiencing high load. Additionally, many of these products also have sophisticated
analytics capabilities that allow analysts to drill down and understand what is happening at a microservices or application level.
A UNIFIED OBSERVABILITY APPROACH

In order to achieve an even deeper level of observability, every unit of data needs to be contextualized. So every metric, trace,
and log captured needs to have the complete context of what is happening in the system. Additionally, developers need to
be able to monitor the whole application behavior, from the start of the delivery pipeline through the deployment. Tools that
integrate seamlessly into the application environment are crucial to automate this whole observability journey.
This unified observability approach has an exciting promise: to provide a deeper correlation between the three pillars and help
teams define and track metrics that are important for their business requirements. 
The state of Kubernetes observability is constantly evolving, so it's important to stay up to date on the latest trends. This
includes learning about the different types of monitoring tools that are available and choosing the best one for your needs. 
Popular Tools for Cloud-Native Observability

In the last few years, the implementation of observability has become more accessible, as there are many open-source tools
available that help developers introduce different aspects of observability into their systems.
Let's look at the most popular ones:
Table 1
POPULAR CLOUD-NATIVE OBSERVABILITY TOOLS
Tool Description Advantages Disadvantages
Prometheus The most adopted open- • Real-time metrics and data collection • Doesn't have built-in long-term storage
source observability tool • Grafana integration • Rudimental anomaly detection
for event monitoring and
alerting • Visualization for containerized • Handles only metrics, not logs or traces
applications • Has challenges with horizontal scaling
OpenTelemetry This project standardizes • Expands the fourth pillar called • Doesn't provide backend storage
how you collect and send "profiling" to help better understand • Doesn't have a visualization layer
telemetry data to a back- performance
end platform, such as • Doesn't provide a storage solution
• The Kubernetes Dashboard can be used
Prometheus or Jaeger to visually track important info, like
containers and Pods running in clusters
• Metrics about apps and app
deployments running in the Pods, as
well as resource utilization
Jaeger A popular choice for • Makes debugging easier by using • Offers limited back-end integration
Kubernetes tracing; tracing to analyze root causes and
its main purpose is to monitor distributed transactions
monitor microservice-
based systems and
collect data in storage
back ends of your choice
cAdvisor A Kubernetes built-in • Collects data on a node level, such • Doesn't offer rich functionality, as it
monitoring feature that as resource usage and performance collects only basic utilization metrics
exists by default on characteristics, and displays them on a • Doesn't offer any long-term storage
every node web-based interface
• No deep analysis or trending metrics
• Handles only metrics, not logs or traces

Conclusion
Cloud-native observability has emerged as an important part of managing Kubernetes deployments in the cloud. By providing
visibility into the performance of applications running on Kubernetes, cloud-native observability has helped to improve
system performance, reliability, and security. With the advancements in open-source tools, we are lowering the barriers of
implementing observability for organizations. Current offerings and standards will mature and become more established by
focusing on the outcomes rather than technical implementation.
Additionally, the trend of moving applications to Kubernetes is likely to increase the demand for observability solutions in the
future. Kubernetes helps developers ship their apps to the cloud in a containerized matter, which is now deeply connected to
the cloud-native way of working. More than just a technology, Kubernetes becomes a business decision that drives the value
for the company that incorporates it, because it means that you develop and deliver apps in a cloud-native manner, making
you ready for innovation.
Kubernetes is most powerful when properly managed and monitored. That's why it's important to think about observability
from the start. Implementing the right tools and standards for cloud-native observability can save your business valuable time
and resources because it will help the organization detect incidents and resolve them in a timely manner.
Marija Naumovska, Co-Founder & Technical Writer at Microtica

@marulka on DZone | @mmarulka on Twitter | microtica.com
As a co-founder of Microtica, Marija helps developers deploy their applications on the cloud in a self-
service manner. She's a Software Engineer with about 10 years of experience, who now works as a product
person and technical writer full time. She writes about cloud, DevOps, GitOps, and Kubernetes topics.

Scaling… to K8s Mess

By Daniel Stori, Software Development Manager at AWS
Daniel Stori, Software Development Manager at AWS

@Daniel Stori on DZone | @dstori on LinkedIn | @turnoff_us on Twitter | turnoff.us
Passionate about computing since writing my first lines of code in Basic on Apple 2, I share my time
raising my young daughter and working on AWS Cloud Quest, a fun learning experience based on 3D
games. In my (little) spare time, I like to make comics related to programming, operating systems, and
funny situations in the routine of an IT professional.

Architectural Patterns for

Microservices With Kubernetes
An Evolution in Microservices Architecture
By Ray Elenteny, Solution Architect at SOLTECH, Inc.
For some time, microservices have drawn interest across the architecture and software engineering landscape, and now,
applications comprised of microservices have become commonplace. So what exactly is the definition of a microservice?
That is somewhat of a loaded question as there is plenty of debate on granularity, segmentation, and what designates a
microservice. For the purposes of this discussion, a microservices-based architecture is segmenting an application's units of
work into discrete, interoperable components. This is a broad definition, but it is workable in that it identifies two foundational
microservice concepts: discrete and interoperable.
Along with the technical and business benefits, a microservices-based application architecture brings its own set of challenges.
These challenges have been met with solutions ranging from new architectural patterns to the evolution of tech stacks
themselves. Kubernetes has become one of the technologies in the tech stack evolution. Deploying microservices using
Kubernetes enhances and enforces key principles and patterns while offering additional benefits.
It's an Evolution, Not a Revolution

As with any technical evolution, taking the next step improves upon what has already been shown to be successful while
removing barriers to adoption or execution. Kubernetes is not going to address all microservices challenges, but it does address
several pain points.
BEST PRACTICES REMAIN

In many cases, the development and packaging of microservices destined for Kubernetes deployment is no different than a
non-Kubernetes deployment. Non-Kubernetes deployments include bare metal servers, virtual machines, and containerized
applications. Applications already packaged for containerized deployment make the step to adopt Kubernetes-managed
microservices straightforward.
• All key microservices patterns, development, and deployment best practices are applied.
• Application tech stacks and components are unchanged.
• Continuous integration/continuous delivery (deployment) systems remain intact.
• Operating system platforms and versions can be tightly controlled.
DIFFERENCES
The differences between Kubernetes and non-Kubernetes microservices architectures focus less on the task performed by the
microservices and more on the deployment of non-functional requirements. Satisfying non-functional requirements is not a
new concept introduced by Kubernetes or even by a microservices architecture. However, through a combination of leveraging
the services offered by Kubernetes itself as well as defining cross-cutting application support services, supporting many non-
functional requirements becomes transparent to an application. The following are two examples.
KUBERNETES INGRESS
A Kubernetes Ingress is an example of a configurable service that auto-configures external access to microservices. When a
microservice is deployed, it can define whether and how it is to be externally accessed. If a microservice specifies that it is to
be externally accessible, the Ingress services within the Kubernetes cluster automatically configure external access, including
details such as virtual host definitions and SSL certificates.

Figure 1: An Ingress definition supporting two services
Here, a Kubernetes Ingress accepts HTTP(S) requests external to the Kubernetes cluster and, based on the request path, routes
requests to specific services within the cluster.
OPERATORS
Kubernetes Operators are a Cloud Native Computing Foundation (CNCF) specification outlining a pattern that supports
cross-cutting application services. They behave similarly to a Kubernetes Ingress in that a service is auto-configured based on
application specification. The primary difference is that Kubernetes Operators present an abstraction where any type of service
is automatically configured to extend the behavior of a Kubernetes cluster. There are Kubernetes Operators that connect
applications to logging and metrics systems with the application knowing little of the specifics regarding those systems'
implementation. There are also Kubernetes Operators that will build and deploy complete database instances.
Figure 2: Kubernetes Operator flow
In the diagram above, an application requests that a service be made available for its use. The Kubernetes Operator monitors
and watches for requests. When a request is made, the Kubernetes Operator instructs the Kubernetes cluster to deploy or
configure a cross-cutting service specific to the application's request.
ABSTRACTIONS
Kubernetes provides and supports abstractions over many systems required to satisfy non-functional components. Successful
Kubernetes microservices architectures are comprehensive beyond application architecture, considering a strategy to not only
address interoperability across microservices but coordination with common services.
Applying Kubernetes Constructs to a Microservices Architecture

Kubernetes deploys container-based applications; this implies that an artifact of a microservice build and packaging process is
a Docker (or suitable alternative) image. In Kubernetes, the basic deployment unit for an image is a Pod. Often there is a one-
to-one relationship between a deployed image and a Pod. However, Kubernetes Pods can support multiple deployed images
within a single Pod. While the deployed containers do not share a file system, they can reference each other using localhost.

Within a Kubernetes cluster, deployed Pods can provide their services to other Pods. This is like a deployed microservice
on bare metal or a virtual machine, although this deployment doesn't provide access to the Pod's service from an external
resource, high availability, or scalability. As discussed, Kubernetes helps applications meet non-functional requirements. A
general rule of thumb is when "-ility" is used to describe a function, it often means a non-functional requirement. Using high
availability and scalability as example, Kubernetes provides these with relative ease. There are a few Kubernetes constructs that
support these functions. Two are presented here: Services and Deployments.
Kubernetes provides a construct called a Service. A Kubernetes Service specifies ports that a microservice wishes to expose
and how they are to be exposed. Services provide two powerful features. First, a Kubernetes Service integrates with the internal
Kubernetes DNS service to provide a consistent host name by which the microservices are accessed within the Kubernetes
cluster. In addition, if there are multiple instances of the same microservice Pod, a Kubernetes Service can act as a load
balancer across the Pod instances, providing high availability.
While Pod instances can be individually deployed, manually monitoring their status is impractical. A common pattern for
adding automation to Pod "-ilities" is Kubernetes Deployments. Kubernetes Deployments specify details surrounding Pod
definitions and provide several features that support the production deployment of microservices, including:
• The number of replicas to be maintained

• Updating the state of declared Pods
• Rollback to earlier versions
• Scaling up or down the number of Pods
With Pod, Service, and Deployment definitions, a solid microservices architecture is in place. In this microcosm, one piece
remains — that is, auto-scaling. With Deployments, scalability is available, but like direct Pod deployments, they are manually
controlled. The final component to this architectural pattern is using a HorizontalPodAutoscaler to automatically scale the
number of Pod instances based on certain criteria (e.g., CPU usage).
This example demonstrates how Kubernetes can take any containerized microservice and, using Kubernetes constructs, satisfy
the critical non-functional requirements that most applications require. Assembling the patterns discussed here, the following
diagram presents a high-level visual of a Kubernetes microservices deployment pattern:
Figure 3: Putting it all together
The diagram portrays two microservices, "greensvc" and "bluesvc." Each microservice utilizes a Kubernetes Service to expose
its functionality. In addition to providing high availability by load balancing multiple Kubernetes Pods per microservice, the
Kubernetes Service maps expose Pod ports to port 80. The definition of a Kubernetes Service also creates DNS entries internal
to the Kubernetes cluster (greensvc.ns.cluster.local and bluesvc.ns.cluster.local) that can allow microservices to interoperate.
Both microservices are exposed outside the Kubernetes cluster through a Kubernetes Ingress. The configured ingress routes
incoming requests to their respective services.

Microservices Deployment Patterns
Kubernetes provides many constructs and abstractions to support service and application Deployment. While applications
differ, there are foundational concepts that help drive a well-defined microservices deployment strategy. Well-designed
microservices deployment patterns play into an often-overlooked Kubernetes strength. Kubernetes is independent of runtime
environments. Runtime environments include Kubernetes clusters running on cloud providers, in-house, bare metal, virtual
machines, and developer workstations. When Kubernetes Deployments are designed properly, deploying to each of these and
other environments is accomplished with the same exact configuration.
In grasping the platform independence offered by Kubernetes, developing and testing the deployment of microservices can
begin with the development team and evolve through to production. Each iteration contributes to the overall deployment
pattern. A production deployment definition is no different than a developer's workstation configuration. This pattern provides
a level of validation that is difficult to reproduce in any previous pattern and can lead to rapid maturity of an application's
delivery cycle.
The Kubernetes ecosystem offers tools that support these patterns. The most predominant tool is Helm, which orchestrates
the definition, installation, and upgrade of Kubernetes applications. It's through tools such as Helm that the same deployment
definition can be executed across multiple runtime environments by simply supplying a set of parameters specific to a runtime
environment. These parameters don't change the deployment pattern; rather, they configure the deployment pattern to meet
the runtime environment (e.g., configuring the amount of memory to allocate to a process).
To learn more about Helm charts, check out the article, "Advanced Guide to Helm Charts for Package Management in
Kubernetes," on page 42 in this Trend Report.
Microservices Deployment in Kubernetes Makes Sense

Deploying microservices in Kubernetes is an evolution of microservices architectures. Kubernetes addresses many pain
points and challenges in developing and deploying microservices-based applications. Being an evolution implies that it's not
a revolution. It's not a rewrite. When designing microservices, in many ways, Kubernetes addresses the question that needs
to be answered. Rather than waiting, good Kubernetes design and deployment patterns encourage tackling non-functional
requirements early in the development process, leading to an application that will mature much faster.
Whether it's Kubernetes or a different deployment platform, the same issues that need to be considered will need to be
addressed up front or later. In software engineering, it's almost always best to consider issues up front. Kubernetes directly
helps in addressing many microservices architectures and deployment challenges.
Ray Elenteny, Solution Architect at SOLTECH, Inc.

@rbetae on DZone | @ray-elenteny on LinkedIn
With over 35 years of experience in the IT industry, Ray thoroughly enjoys sharing his experience by
helping organizations deliver high-quality applications that drive business value. Ray has a passion for
software engineering. Over the past 10 years or so, Ray has taken a keen interest in the cultural and
technical dynamics of efficiently delivering applications.

Advanced Guide to Helm

Charts for Package
Management in Kubernetes
By Yitaek Hwang, Software Engineer at NYDIG
Helm is undoubtedly one of the most popular and successful open-source projects in the Kubernetes ecosystem. Since it was
introduced at the inaugural KubeCon in 2015, Helm's usage has steadily grown to solidify its status as the de facto package
manager for Kubernetes. It graduated from the Cloud Native Computing Foundation in 2020, and in the CNCF Survey 2020,
more than 60% of the survey respondents reported Helm as their preferred method for packaging Kubernetes applications.
This rates significantly higher than Kustomize and other managed Kubernetes offerings.
Figure 1: Kubernetes package management preference survey results
Image credit: "CNCF Survey 2020," Cloud Native Computing Foundation
Given Helm's popularity and ubiquity, it's important to understand the architecture and usage patterns to utilize the tool
effectively. In this guide, we'll break down the key concepts underlying Helm's structure and walk through some best practices.
Helm Structure
Helm packages Kubernetes applications into charts, which is a collection of templated Kubernetes manifest files. Each chart
contains the following content:
example-chart/
├── .helmignore
├── Chart.yaml
├── values.yaml
├── charts/
└── templates/
└── tests/
...

Let's dive deeper into each file:
• .helmignore – Defines files or patterns to ignore when packaging up the Helm chart (e.g., .vscode, .git, secret files).
• Chart.yaml – Contains top-level metadata about the Helm chart including the name, version, and other dependencies
if the chart bundles multiple applications together. Examples of dependent charts include packaging Zookeeper with
Kafka or grouping Prometheus and Grafana together.
• values.yaml – Sets default values for all templated portions of the chart under templates directory. These default values
can be overridden via –values or –set flags.
• charts – Stores tar files of dependent charts if it was specified in Chart.yaml.

• templates – Holds all the Kubernetes manifest files that define the application, including but not limited to Deployment,
Services, autoscaling, ConfigMaps, and Secrets. Along with core Kubernetes files, it can also hold template helpers,
NOTES.txt, and test files.
Underneath the hood, Helm uses Go templates to coalesce values into the corresponding template files. This means that it's
important to understand YAML syntax as well as Go's data types to create, update, or use a Helm chart.
HELM LIFECYCLE
Once the chart is created, it is ready to be deployed into a Kubernetes cluster. The Helm CLI uses install or upgrade
commands to trigger a new release. Each release is an isolated, deployed artifact of a versioned, templated chart. This means
that it is possible to have multiple releases of the same chart as long as the underlying values do not conflict. For example, a
PostgreSQL Helm chart can be deployed with the same configurations to two different namespaces.
Also, Helm keeps a history of each release, allowing easy rollbacks. To see the release history, run helm history <name-of-
release>. Then to roll back, use helm rollback <name-of-release> <version>. While these CLI commands are useful for
development, these will most likely be controlled by a more robust CD system in production.
Helm Best Practices

Now that we have a good understanding of Helm's structure and lifecycle, we'll review some best practices and useful tips to
interact with Helm charts.
BOOTSTRAP CHARTS WITH HELM CLI

Helm provides an excellent skeleton to bootstrap a new Helm chart via the helm create command. Instead of handcrafting
individual Helm files from scratch, take advantage of the create command to get default manifest files (e.g., deployment.yaml,
hpa.yaml, ingress.yaml, service.yaml, and serviceaccount.yaml) as well as templates to provide unique names and
annotations.
CHECK FOR ERRORS WITH HELM CLI

Helm also provides several tools to validate the chart before deploying. YAML and Go template syntax can be difficult to debug,
especially when it's due to simple indentation or empty value errors.
1. helm lint – Examines the chart for syntax issues such as poorly formatted YAML indentation or missing values. Running
this command with the –debug flag can provide more information.
2. helm template – Locally renders the templates using provided values. This can be useful to see if the override values are
correctly inserted into the manifest templates. To focus on a single template, use the -s flag: helm template my-chart -s
templates/deployment.yaml.
3. --dry-run – Lint and template commands can only catch templating issues via static analysis. To catch Kubernetes-
related errors (e.g., using a deprecated Kubernetes API version), run the --dry-run command before invoking the
upgrade or install commands.
For advanced guides on template functions, flow controls, and other debugging tricks, refer to the Helm documentation
section under "Chart Template Guide."
PACKAGE WITH SUBCHARTS

As mentioned in the Helm structure section, sometimes it makes sense to package multiple applications together into a single
Helm chart. This pattern is useful to group related applications or databases together into a single deployable unit. Common
use cases include packaging PostgreSQL and Redis with a Django app or bundling all the components of the ELK/EFK stack

together. Another way to use subcharts is to bundle common helper templates to add annotations, labels, or default values. A
good example can be found with Bitnami's charts:
apiVersion: v2
appVersion: "3.2.1"
dependencies:
- condition: zookeeper.enabled
name: zookeeper
repository: "https://charts.bitnami.com/bitnami"
version: 10.x.x
- name: common
repository: "https://charts.bitnami.com/bitnami"
tags:
- bitnami-common
version: 2.x.x
name: kafka
version: 18.2.0
USE ANNOTATIONS
Since Kubernetes only watches changes in the deployment spec, by default, changing the contents of a ConfigMap or a Secret
that's mounted to a Pod does not automatically trigger a rolling update. This may be unexpected behavior for users expecting
a restart with updated values when invoking an upgrade command.
To automatically roll deployments, you can use tools like Reloader or add a checksum annotation in the deployment spec:
kind: Deployment
spec:
template:
metadata:
annotations:
checksum/config: {{ include (print $.Template.BasePath "/config.yaml") . | sha256sum }}
Annotations are also useful to define hooks for jobs that should run before or after an install or upgrade. Common use cases
may include running a Flyway migration or running clean-up jobs. While this can be achieved with other CD tools, Helm hooks
provide a native way to alter release lifecycle behavior:
apiVersion: batch/v1
kind: Job
metadata:
annotations:
"helm.sh/hook": post-install
"helm.sh/hook-weight": "-5"
"helm.sh/hook-delete-policy": hook-succeeded
Limitations of Helm
While Helm comes with a suite of features to make packaging and releasing Kubernetes applications easier, it's important
to note that Helm is simply a package manager — it is not a fully featured CD tool. In other words, organizations should not
rely on Helm alone for an end-to-end Kubernetes CI/CD tool. Instead, Helm should integrate with more robust tools such
as ArgoCD, JenkinsX, or Spinnaker to keep deployments consistent. Also, since Helm uses plaintext YAML as the primary
templating interface, Secrets and non-YAML files are not handled very well. This means Secrets or configuration binaries (e.g.,
public keys, .dat files) can be better handled outside of Helm via another CI/CD tool.
Finally, while subcharts are useful for organizing related components together, deeply nested subcharts can cause templating
issues. If the application truly requires multiple nested subcharts, look for other deployment patterns such as ArgoCD's
ApplicationSet to group multiple Helm charts together instead.

Final Thoughts
Helm is a powerful tool that has stood the test of time. Compared to verbose, vanilla Kubernetes manifest files, Helm allows
for advanced templating techniques to package Kubernetes applications into deployable units. In this guide, we went over
the high-level Helm structure as well as some best practices to use Helm effectively. We then walked through some of the
limitations of Helm — mainly the fact that it should not be used as a standalone CD tool.
Nonetheless, with a wide selection of public, production-ready Helm charts, it's easier than ever to download, modify, and
install various Kubernetes components.
Yitaek Hwang, Software Engineer at NYDIG

@yitaek on DZone | @yitaekhwang on LinkedIn | yitaekhwang.com
Yitaek Hwang is a software engineer at NYDIG working with blockchain technology. He often writes about
cloud, DevOps/SRE, and crypto topics.

An Overview of CI/CD
Pipelines With Kubernetes
Exploring the Approaches, Tools, and Benefits of
Implementing CI/CD in a Kubernetes Ecosystem
By Sudip Sengupta, Technical Writer at Javelynn
Automation and cross-functional collaboration are two fundamental features of a DevOps model. As one of the core
components of a DevOps framework, continuous integration and continuous delivery (CI/CD) pipelines speed up application
delivery by automating various stages of the software development lifecycle. Although Kubernetes inherently automates
the delivery, scaling, and management of applications by deploying and orchestrating them in flexible, portable containers,
adopting a CI/CD-based workflow brings a plethora of additional features by offering an iterative cycle of configuration merges
and testing.
In this article, we discuss CI/CD approaches in a Kubernetes ecosystem, best practices for implementing an efficient CI/CD
framework, and popular open-source CI/CD tools.
CI/CD Pipelines in a Kubernetes Ecosystem

Kubernetes provides a combination of deployment objects, rollout strategies, and resource controllers to help automate the
build, test, and integration processes of a CI/CD pipeline. Beyond container-based deployments, Kubernetes powers modern
CI/CD pipelines using efficient orchestration
algorithms that schedule batch jobs, enable
gradual rollouts, and autoscale workloads. With Figure 1: Pull-based CI/CD pipeline
containerized deployments, DevOps teams

can efficiently automate the design, testing,
coding, integration, delivery, and deployment
of applications for comprehensive automation.
KUBERNETES CI/CD APPROACHES

Kubernetes supports two approaches to
implement a CI/CD pipeline that essentially
define how configuration changes are
implemented in a workflow. These include
pull-based pipelines and push-based pipelines.
PULL-BASED PIPELINE
In a pull-based CI/CD pipeline, code and
infrastructure changes are applied within
the cluster. This architecture relies on an
agent (often referred to as a controller) within
the cluster that scans the cluster state and
compares it with a declarative desired state. In
instances where there is a difference between
the desired configuration and cluster state, the
controller automatically applies changes.

To help further understand how a pull-based CI/CD pipeline can support your cloud-native workloads, let us learn some of its
advantages and disadvantages:
Table 1
PULL-BASED CI/CD PIPELINE
Advantages Disadvantages
• It ensures continuous security by removing manual overheads to grant admin • It only works for Kubernetes deployments.
access for every cluster change. • In a multi-cluster framework, each Kubernetes
• It is easier to scale identical deployments since the controller automatically cluster should adopt the pull-based framework.
applies changes to the cluster without requiring manual efforts.
PUSH-BASED PIPELINE
This is an agentless pipeline architecture that uses an external system to trigger deployments. In a push-based pipeline,
configuration changes are committed in a repository, which then updates the environment by adding commits to a pipeline
that was already successfully executed.
Figure 2: Push-based CI/CD pipeline
In spite of its quick implementation and popularity, organizations should assess a push-based pipeline's use cases before
choosing it for their cloud-native workloads. Listed below are some of its advantages and disadvantages:
Table 2
PUSH-BASED CI/CD PIPELINE
Advantages Disadvantages
• It is simpler to implement when compared to a pull-based pipeline. • It requires external admin access to the
• It offers flexibility and can be applied to environments other than Kubernetes. CI/CD pipeline.
• Push pipelines standardize the deployment of workloads for both cloud and • It compromises security as the architecture
traditional environments, making it ideal for hybrid/multi-cluster models. relies on opening of the cluster firewall for
external services.
CONSIDERATIONS FOR KUBERNETES CI/CD PIPELINES

Developing and deploying applications on a Kubernetes-based CI/CD pipeline relies on an autonomous framework of various
Kubernetes tools, services, and best practices that ensure security, high performance, and agility. Some key considerations for
implementing efficient Kubernetes CI/CD pipelines are automated testing, validation, and rollbacks.
AUTOMATED TESTING
Detecting errors and defects during early stages of a workflow helps enhance application agility. As a recommended design,
the test environment should match the production environment to streamline the build, test, and deploy cycles across
different environments. Continuous, automated testing also helps create sustainable application builds that ensure the code is
readable, debuggable, and maintainable.

VALIDATION
Since a CI/CD pipeline may have multiple builds deployed simultaneously, it is crucial to have an automated validation process
to track points of failure in large-scale deployments. With appropriate validation, the pipeline integrates code changes
seamlessly, thereby reducing vulnerability detection overhead.
ROLLBACKS
Rollbacks ensure code sanity by restoring a previous, stable deployment if there are flaws with a new commit/version. As one of
the most critical considerations of designing a workflow, it is important to adopt a planned roll-back strategy that safeguards
the availability of workloads in production.
BENEFITS OF CI/CD PIPELINES IN A KUBERNETES ECOSYSTEM

Advantages of using Kubernetes-based CI/CD pipelines include:
• Shorter release cycles – Automated testing and deployment eliminate delays in pushing changes to production. Iterative
CI/CD validation also avoids code merge collisions, subsequently expediting the frequent patching and updating of
workflows in production.
• Autoscaling – Containerizing applications makes it easy to scale and replicate workloads with changing traffic demands.
Kubernetes deployment objects such as ReplicaSet, Deployment, and StatefulSet help automatically spin up replicas of
containerized workloads to ensure efficient load balancing and near-zero downtime.
• Node-usage efficiency – With Kubernetes-based workflows, administrators can set optimum resource limits and
requests, ensuring efficient utilization of server compute, memory, and storage resources.
• Microservices-based deployments – Kubernetes enforces the deployment of applications in portable containers with all
necessary libraries and resources. The framework allows developers to leverage microservices for simpler, cloud-native
application deployment and management.
Popular Open-Source Kubernetes CI/CD Tools

Open-source CI/CD tools are known to offer flexibility in the deployment, scaling, and management of workloads in Kubernetes
CI/CD pipelines. While organizations often tend to adopt the most popular platforms, it is imperative to analyze business and
technical requirements before choosing one. Some popular open-source CI/CD tools include Spinnaker, Jenkins X, and Argo CD:
Table 3
Tool Description Advantages Disadvantages
Spinnaker A multi-cloud CI/CD platform that relies on JSON to • Rapid releases and deployments • Adds operational
implement and configure software delivery pipelines. • Integrated deployment overhead for
The platform implements in-built, production-safe strategies orchestration of
deployment strategies to help deliver composite multiple underlying
applications to their target environments without • Rollbacks can be implemented microservices
having to write any scripts. with a single click
• Allows only immutable
By following an API-based modular architecture, • Out-of-the-box support for deployments
Spinnaker allows DevOps teams to integrate external multi-cloud and hybrid models
services seamlessly for enhanced security, visibility,
and cost efficiency.
Jenkins X Helps accelerate software delivery by creating • Provides automated feedback on • Limited to Kubernetes
previews on pull requests while using Git as the single pull requests and issues clusters and works
source of truth. As the platform provides an official • Relies on Git as a single source purely on GitHub
Kubernetes plugin, it is considered one of the most of truth • Inefficient UI and
adopted CI/CD tools for Kubernetes workloads. dashboard view
Argo CD A declarative, Kubernetes-centric continuous delivery • Kubernetes-native deployments • Being dependent on a

tool that fetches commits made in a centralized • Enforces portability and flexibility pull-based mechanism,
repository and applies them to the production cluster. it requires a longer
• Uses standard, declarative implementation cycle
The platform relies on an agent that is installed patterns for application delivery
directly within the cluster to implement pull-based • Does not offer
GitOps for managing application updates and continuous integration
infrastructure configuration. out of the box

Best Practices for Implementing CI/CD With Kubernetes
Although approaches toward adoption of a DevOps model may differ with varying use cases, let's discuss some commonly
recommended practices for implementing efficient Kubernetes CI/CD pipelines.
DILIGENTLY CHOOSING AN IDEAL ENVIRONMENT

Developers should diligently choose the appropriate version of Kubernetes as this further defines the choice of CI/CD tools
and methodologies. Depending on the complexity of workloads deployed, developers should also be sure to choose the ideal
setup between managed, self-hosted, cloud-based, or on-premises since these influence critical aspects of a CI/CD workflow
including scaling, security, and cost.
SECURE SECRETS
Cluster Secrets and configurations are recommended not to be included within container images or Pod definitions. As an
alternative, Secrets should be stored in an encrypted vault or defined as Kubernetes Secret objects. Additionally, configuration
settings are recommended to be declared in a ConfigMap, which is then mounted to the container. The approach helps define
specific configurations for each environment without exposing data in the container.
UTILIZING GITOPS WORKFLOWS

GitOps is a modern integration workflow that ensures predictability, visibility, and ease of use. Using a single source of truth
simplifies CI/CD-based application delivery and infrastructure control, while streamlining code integration and eliminating
redundancy.
Summary
Kubernetes continues to maintain favorable adoption in modern DevOps workflows as it offers seamless scalability and
flexibility in modern application delivery. When configuring clusters at scale, managing a Kubernetes ecosystem can quickly
become challenging. Overcoming such complexities often requires a combination of recommended practices, diligently
designed architecture, and efficient tools that help achieve agility, scalability, and comprehensive automation. Although
Kubernetes powers CI/CD workflows, open-source platforms are also recommended to reduce testing and deployment
overhead toward configuring native pipelines.
Sudip Sengupta, Technical Writer at Javelynn

@ssengupta3 on DZone | @ssengupta3 on LinkedIn | www.javelynn.com
Sudip Sengupta is a TOGAF Certified Solutions Architect with more than 17 years of experience working for
global majors such as CSC, Hewlett Packard Enterprise, and DXC Technology. Sudip now works as a full-
time tech writer, focusing on Cloud, DevOps, SaaS, and cybersecurity. When not writing or reading, he's
likely on the squash court or playing chess.

An Assessment of Kubernetes
and Machine Learning
Advantages for ETL Pipelines and ML Models
By Eric Goebelbecker, DevRel at HitSubscribe
Kubernetes and machine learning (ML) are a perfect match. As the leading container orchestrator, Kubernetes' scalability and
flexibility make it the perfect platform for managing extract-transform-load (ETL) pipelines and training ML models. That's why
there's a thriving ecosystem for running ML tasks on Kubernetes. Let's look at how Kubernetes is uniquely suited for supporting
machine learning, how you can use it to make your ML pipelines run faster and better, and some of the most popular
Kubernetes tools for ETL and ML.
ETL and ML Pipeline Challenges

When it comes to building ML and ETL pipelines, engineers face many common challenges. While some are related to data
access and formatting, others are the result of architectural problems.
• Resource management – Data volumes change over time and pipelines need a cost-effective way to manage resources.
• Dependency management – Pipelines need to execute tasks in the proper order and with the correct timing.
• Platform upgrades – Keeping up with security patches and vendor releases is a challenge for any platform engineer.
• Code upgrades – As data changes, so do the requirements for the systems that process it. Pipelines must be built on a
platform that makes it easy to deploy and test new code.
Benefits of Kubernetes and ML

Kubernetes addresses many of the platform problems that data engineers wrestle with when they need to build or upgrade
ETL and ML pipelines. By combining containers with automatic scaling, support for all major cloud providers, and built-in GPU
support, it makes deploying code easier and allows engineers to focus on their core capabilities.
KUBERNETES HAS THE POWER TO SUPPORT ML

Kubernetes automates the deployment, scaling, and management of containers. This makes it the preferred platform for
running ETL pipelines and ML algorithms. Automated scalability means that the system can adjust to differing loads on the fly
without your intervention. Whether it's an unusually large dataset or fewer training sessions than normal, Kubernetes will scale
your cluster up or down to compensate. This means less engineering time bringing newer systems online and lower cloud
costs because unused systems stay shut down.
The advent of cloud-based Kubernetes systems like EKS and GKE has made building and managing Kubernetes clusters easier.
But that doesn't mean you want to build one for every project. Kubernetes namespaces make multi-tenancy easy. You can
share a single cluster across multiple projects and teams. You can even create namespaces for different versions of projects
when you want to test new containers or algorithms.
Tracking versions and maintaining code consistency across different servers has always been a headache. Containers solve this
issue by abstracting the infrastructure for you. You can target your model code for a single container runtime and Kubernetes
can execute it anywhere. When it's time to update your code, you do it once in the container. You can even deploy your model
against different code versions in the same cluster.
Kubernetes has experimental support for scheduling AMD and NVIDIA GPUs. You can attach GPUs to containers that need
them, and Kubernetes will manage access and ensure that they're not overallocated. So the scalability Kubernetes provides you
for training algorithms extends to advanced GPU hardware, too.

KUBERNETES FACILITATES ETL
ETL pipelines are, by nature, modular workflows. The name tells you that there are at least three steps:
1. Extract data
2. Transform data into a usable format
3. Load data into storage for the training models that use it
Containers and Kubernetes make it easy to break down these processes into reusable components. And just like with models,
the steps that run in containers are abstracted from the underlying hardware. Kubernetes also simplifies managing parallel
pipelines for multiple datasets and establishing dependencies between steps in a pipeline.
Let's look at a pipeline in Figure 1 that has more than one data source. It needs to download two datasets, transform them, then
collate them before loading.
Figure 1
The pipeline needs to extract data from Sources A and B. Then it needs to transform them in independent processes. Only
when both transformations are done can the collation step begin. Since each of these steps is a container, Kubernetes
manages starting each one in order and ensures that the transform step doesn't load until both transform steps have
completed their work. This is a straightforward example. You can manage more complicated pipelines with Kubernetes
scripting tools like Airflow or Argo Workflows, which make it easy to run pipelines as directed acyclic graphs.
Finally, Kubernetes manages secrets for you, so it even makes pipelines that need to access data via logins or with PKI
more secure.
KUBERNETES FACILITATES MLOPS

Machine learning operations (MLOps) is a collection of machine learning tools and strategies. They make a data scientist's life
easier by formalizing and automating the running of models. Kubernetes simplifies MLOps. A container orchestration system
makes it possible for a team to run more than one version of a model in parallel. So instead of waiting for a single experiment to
complete before trying the next, you can deploy more than one container in parallel and collect your results as they complete.
We've already mentioned how Kubernetes brings scalability to ML, but it bears mentioning again. With Kubernetes, you can
drive models at a massive scale, and when you're done, you can destroy the nodes to cut costs.
KUBERNETES ML TOOLS
Kubernetes' native ability to orchestrate containers originally drew data scientists to it, but since then, teams have developed
powerful tooling to make using it even better. Kubeflow is an open-source ML toolkit for deploying ML workflows on
Kubernetes. It brings best-of-breed ML tools to Kubernetes in a way that makes them easier to use while taking advantage of
container orchestration.
With Kubeflow, you can:
• Manage interactive Jupyter notebooks to experiment with models and deploy them to the cluster
• Train TensorFlow models and configure them for GPUs
• Build custom Kubeflow pipelines
• Export TensorFlow containers to the cluster

There are also several workflow orchestrators like the Argo Workflows and Airflow tools mentioned above. They make creating
an ML pipeline with parallel processing and dependencies easy thanks to simple script languages.
Kubernetes ML Platforms
The list of tools that aid in running ML pipelines on Kubernetes doesn't stop there. There are many open-source and
commercial tools that you can use to manage ETL and models on your K8s clusters:
Table 1
Open Source or Programmatic

Name Commercial and/or GUI Description
Kubeflow Open source Both Collection of ML tools for managing and running models on a cluster
Open MLOps Open source Both Collection of Terraform scripts for setting up ML tools on a cluster
Bodywork Open source Programmatic Python-based tools for deploying and using ML pipeline tools on a cluster
Apache Airflow Open source Both Scripting platform for orchestrating workflows
Argo Workflows Open source Both Scripting platform for orchestrating workflows
AWS Sagemaker Commercial Both AWS ML platform that will work with AWS EKS
Azure ML Commercial Both Azure ML platform that will work with Azure Kubernetes
Conclusion
We've covered how Kubernetes makes running ML pipelines faster, easier, and more cost efficient. With containers, you can
create ETL jobs and models that run anywhere. With Kubernetes, you can run those containers in parallel and at scale. We also
took a brief look at some of the open-source and commercial tools available to make setting up your ML jobs even simpler.
Now that you're familiar with the power Kubernetes can bring to your ML and ETL jobs, we hope you'll integrate it into your
next ML project.
Eric Goebelbecker, DevRel at HitSubscribe

@egoebelbecker on DZone and Twitter | @ericgoebelbecker on LinkedIn | ericgoebelbecker.com
For nearly 30 years, Eric worked as a developer and systems engineer at various Wall Street market data
and trading firms. Now he spends his time writing about technology and science fiction, training dogs,
and cycling.

Kubernetes Cost Optimization

A Look at Available Tools and Best Practices
By David Snatch, Consultant at Kontinuous Snatch, Inc.
Day 2 for Kubernetes is here. Many organizations are running their workloads in Kubernetes, and if not, they are planning to do
it soon. However, after adopting Kubernetes in your organization, one of teams' main concerns is often about how to optimize
the costs of their infrastructure. In this article, you will discover best practices to help optimize infrastructure costs for your
Kubernetes clusters and the tools available to help you reduce the monthly bill (mainly in the cloud). Additionally, since it's not
only about infrastructure, you will also find recommendations for developing your applications.
Best Practices for Optimizing Costs

Kubernetes and its ecosystem have been growing exponentially in recent times. Think of it like Legos — there are a lot of pieces
available, and you simply need to connect all the points to make the most of Kubernetes' infrastructure. And the beauty about
optimizing costs is that it will allow you to reduce the monthly bill as well as help you be more efficient with the budget you
already have. So let's start talking about practices you need to consider, like a prerequisite to start optimizing infrastructure.
MAIN CHARACTERISTICS OF AN APPLICATION

Kubernetes does a great job at scheduling your applications in the cluster; however, if your application doesn't comply with
certain prerequisites, deploying them to Kubernetes won't help much. For example, to make applications fault-tolerant, you
need to gracefully manage the SIGTERM signals in containers. It's important we don't forget that containers are immutable
and volatile, meaning applications need to be stateless and fault tolerant. Even though support for stateful applications has
matured significantly, you have more room to optimize infrastructure when applications are stateless.
Fault tolerance is important because Kubernetes nodes and Pods might come and go, mainly because of the elasticity feature
when scaling in or scaling down, so applications need to gracefully manage those interruptions.
Your applications need to have mechanisms to help Kubernetes know that it's healthy. For example, have health-check
endpoints to configure probes within a Pod so Kubernetes can restart or recreate a Pod when needed. And another aspect
many don't consider is the ability to handle SIGTERM signals that the container receives when the kubelet needs to remove
some Pods (again, typically for scale-in events).
IMPLEMENTING AUTOSCALING AT THE APPLICATION LEVEL

Once the application has the key characteristics for elasticity, you need to be precise about which resources they're going to
use. This is because the resource numbers you assign in terms of CPU and memory are the numbers that you'll need to use to
define autoscaling policies. These policies can define how many replicas of a Pod to add or remove.
With horizontal pod autoscaling (HPA), you configure scaling policies to automatically adapt your replicas to the demand
you have. You might need more or fewer Pods. And if you couldn't define a proper number of resources to your application,
you can use a VerticalPodAutoscaler (VPA) to help you set resources based on usage, which ends up optimizing the cluster
infrastructure. You might need bigger or smaller nodes.
As you can see, developing reliable and fault-tolerant applications is the starting point of your journey to optimizing costs
in Kubernetes.
AUTOSCALING INFRASTRUCTURE COMPUTE NODES

Once you're able to configure autoscaling policies for your applications with HPA, at some point you'll end up with Pods
pending to be scheduled because the capacity in the cluster isn't enough. For this, you have projects like Cluster Autoscaler or
Karpenter that will help you to scale out or scale in the nodes in the cluster.

All these projects I mentioned work by deploying a controller in your cluster that is constantly looking for pending Pods and,
based on that, launching new servers. Of course, each of those projects have their own peculiarities that are out of scope for
this article. However, both controllers will help you to do the same with the compute capacity in your cluster.
The controllers will help you to:
1. Choose the right type of server for your workloads, like spot instances, when possible.
2. Add or remove servers based on the number of Pods your applications need. This is helpful, especially for saving costs as
you need an automated way to remove servers that are no longer needed.
USING SPOT INSTANCES

A "spot instance" is unused compute capacity that cloud providers have in their data centers. The three major cloud providers
have spot instances to provision compute nodes in a Kubernetes cluster: AWS' Spot Instances, GCP's Spot VMs, and Azure's
Spot Virtual Machines. Even though all of them have different features for managing these types of instances, the concept is
the same. Instead of letting the compute capacity sit without being used, providers offer large discounts when you use them.
However, when cloud providers need to use that capacity again, those servers will be disrupted, in which case you'll typically
receive a notification. In Kubernetes, this gives you the opportunity to cordon the node and drain the Pods it has.
Using this type of instance is a no-brainer because as long as you have enough capacity in the cluster, your applications
won't be affected — especially if you've applied the practices I recommend in this article. Taking advantage of spot instances
will often enable you to reduce your compute bill in the cloud by 65-90%. It's important to make sure you have the latest
information about how they work for your preferred cloud provider.
Otherwise, you might have too many disruptions and won't be able to get big savings. For instance, in AWS, you no longer
need to bid for a spot instance, and the main recommendation is that you use more than one instance type.
RESCHEDULING PODS IN THE CLUSTER

Pods will come and go in a Kubernetes cluster, and when a Pod needs to be scheduled, Kubernetes makes the best decision at
that time for where to place the Pods. However, later, some Pods might be better placed somewhere else in the cluster. This will
lead you to make better use of the capacity as a whole in the cluster, which, in the end, will help you to optimize compute costs.
Projects like descheduler handle the process of rescheduling Pods from time to time. By rescheduling Pods, Pods will always
land in the cluster where they fit better, helping you to optimize nodes by using the available capacity at its maximum. You
might end up with empty nodes, so nodes will become easier to identify for removal. Moreover, I recommend you configure
Pod disruption budgets to make the rescheduling graceful and avoid disruption in your workloads.
Conclusions
If you're not clear on where to start after reading this information, the pyramid in Figure 1 showcases where you should focus
first. The biggest impact on cutting costs associated with Kubernetes begins at the base of the pyramid — making your
application stateless and fault-tolerant.
Figure 1: Cost optimization pyramid

In this article, my goal was to give you some guidance on where to start and what you should be looking to implement next.
Remember the importance of building stateless and fault-tolerant applications as this is the first step to optimize costs. Then
it's simply a matter of understanding the important bits of how to configure a cluster (depending on the provider) and having
the data to monitor infrastructure costs constantly.
David Snatch, Consultant at Kontinuous Snatch, Inc.

This article was written by David Snatch. David is a cloud architect focused on implementing secure continuous delivery
pipelines using Terraform, Kubernetes, and any other awesome tech that helps customers to deliver results.

ADDITIONAL RESOURCES
Diving Deeper Into

Kubernetes
BOOKS REFCARDS
The Book of Kubernetes: A Comprehensive GitOps for Kubernetes

Guide to Container Orchestration It's safe to say that what DevOps is to the cloud, GitOps is to
By Alan Hohn cloud native. In this Refcard, we'll be taking a look at operating
This hands-on guide will teach you everything models for building cloud-native applications, the key benefits
you need to know about Kubernetes container of GitOps (including stronger security and easier auditing),
orchestration by walking you through modern handling configuration updates with Kubernetes, and more.
architecture, how to create containers, how to manage and
maintain Kubernetes clusters, and more. Included in the Getting Started With Kubernetes
guide are example scripts that will get you started on your Containers weighing you down? Kubernetes can scale them.
containers journey. To build a reliable, scalable containerized application, you
need a place to run containers, scale them, update them,
MULTIMEDIA and provide them with networking and storage. This
updated Refcard covers all things Kubernetes, exploring
DZone Cloud Show: KubeCon 2021 Recap core concepts, important architecture considerations, and
Are your ready for KubeCon 2022? Prepare for this year's how to build your first containerized application.
convention by checking out this recap of KubeCon 2021. This
video hits all of the data and topics you need to refresh your TREND REPORTS
mind, and it will help you understand how Kubernetes has
evolved over the last year. Microservices and Containerization: The Intersection
of Cloud Architectures and Design Principles
Automation for Kubernetes Clusters DZone's 2022 Microservices and Containerization Trend Report
The growth of containerized environments and Kubernetes dives into various cloud architecture practices, microservices
clusters over multiple cloud service providers calls for orchestration techniques, security approaches, and advice
standardized management and automation. In this webinar, about design principles. The goal of this report is to explore the
host Jerry Lee discusses the open-source tool sudoRy, current state of microservices and containerized environments
which was developed for this purpose and executes in order to help developers face the challenges of complex
Kubernetes, Helm, Prometheus, Grafana, and more APIs using architectural patterns in their work.
standardized templates over multiple clusters.
Kubernetes and the Enterprise
TechWorld With Nana Building upon findings from previous years that indicate the
Nana Janashia, a CNCF Ambassador, uses her YouTube technical maturity of containers and container orchestration,
channel to share her expertise on various DevOps topics. From DZone's 2021 Kubernetes and the Enterprise Trend Report
crash courses to specific use cases, Janashia breaks down explores more closely the growing ecosystem and tooling,
everything Kubernetes, Istio, Docker, and beyond. use cases, and advanced strategies for Kubernetes adoption in
organizations and across industries.
Kubernetes Bytes
Ryan Wallner's and Bhavin Shah's podcast
focuses on new and hot topics in Kubernetes.
Featuring developers and engineers from
Speedscale, InfluxDB, Redis, Datastax, and
more, you'll learn from their challenges and experiences in the
cloud native ecosystem.

ADDITIONAL RESOURCES
Solutions Directory
This directory contains Kubernetes and cloud-native tools to assist with deployment, management,
monitoring, and cost optimization. It provides pricing data and product category information gathered
from vendor websites and project pages. Solutions are selected for inclusion based on several impartial
criteria, including solution maturity, technical innovativeness, relevance, and data availability.
DZONE'S KUBERNETES IN THE ENTERPRISE SOLUTIONS DIRECTORY
Company Product Product Type Availability Website
Chronosphere Observability
Chronosphere Cloud native observability By request chronosphere.io/platform
Platform
D2iQ Kubernetes Platform d2iq.com/kubernetes-platform
DKP Essential d2iq.com/products/essential
D2iQ DKP Enterprise Kubernetes platform Trial period d2iq.com/products/enterprise
DKP Edge/IoT d2iq.com/products/edge_iot

2022 PARTNERS
Kaptain AI/ML d2iq.com/products/kaptain
Kasten by Kubernetes backup and cloud-

Kasten K10 Free tier kasten.io/product
Veeam native data management
Lacework Polygraph® Data Platform Data-driven cloud security platform Trial period lacework.com/platform
newrelic.com/platform/kubernetes-
New Relic Pixie Kubernetes observability Free tier
pixie
Cloud Development okteto.com/development-

Okteto Developer experience platform Free tier
Environments environments
Stackwatch Kubecost  Kubernetes cost optimization Free tier kubecost.com
Amazon Elastic Kubernetes Managed service to run Kubernetes

Free tier aws.amazon.com/eks
Service (EKS) on AWS and on-premises
Amazon Web Amazon EKS Distro Kubernetes distribution from AWS Open source aws.amazon.com/eks/eks-distro
Services Amazon Elastic Container
Containers as a Service aws.amazon.com/ecs
Service
Free tier
AWS Fargate Serverless compute for containers aws.amazon.com/fargate
Ambassador Kubernetes-native API gateway getambassador.io/products/api-

Emissary-Ingress Open source
Labs for microservices gateway
Secure image deployment

Anchore Anchore Enterprise and continuous monitoring in By request anchore.com/platform
Kubernetes
Atos Apprenda Cloud Platform Kubernetes-based cloud PaaS By request apprenda.com/platform
aquasec.com/aqua-cloud-native-
Aqua Security Aqua CNAPP Cloud-native security platform By request
security-platform
Declarative, GitOps continuous

Argo CD argoproj.github.io/cd
delivery tool for Kubernetes
Argo Open source
Argo Workflows Kubernetes-native workflow engine argoproj.github.io/workflows
Aspen App Intelligence Automation for app performance

Aspen Mesh By request aspenmesh.io/product
Platform at scale

Open platform for building

Backstage Backstage Open source backstage.io
developer portals
Code transformation to create OCI-

Buildpacks Cloud Native Buildpacks Open source buildpacks.io
compliant containers
Kubernetes as a Service on public

Canonical Kubernetes ubuntu.com/kubernetes
and private clouds
Canonical Open source
Cloud infraustructure deployment
Juju juju.is
and management
Kubernetes cost monitoring and

CAST AI CAST AI Free tier cast.ai
reporting
Chaos Engineering platform for

Chaos Mesh Chaos Mesh Open source chaos-mesh.org
Kubernetes
Software for container workload

Cilium cilium.io
connectivity management
Cilium Open source
Distributed networking and
Hubble github.com/cilium/hubble
security observability platform
Kubernetes progressive delivery

Flagger flagger.app
operator
Cloud Native
Flux GitOps Kubernetes operator fluxcd.io
Computing Open source
Foundation Kubernetes Container orchestration solution kubernetes.io
Notary Artifact management toolset notaryproject.dev
Cloud-native solution for CI docs.cloudbees.com/docs/

CloudBees CloudBees CI By request
delivery at scale cloudbees-ci
Specification for describing event

CloudEvents CloudEvents Open source cloudevents.io
data in a common way
CNI CNI Networking for Linux containers Open source cni.dev
Cockroach Distributed SQL database built

CockroachDB Free tier cockroachlabs.com
Labs for Kubernetes
Container lifecycle management

containerd containerd Open source containerd.io
on host systems
Contour Contour Kubernetes ingress controller Open source projectcontour.io
DNS-based Kubernetes service

CoreDNS CoreDNS Kubernetes Open source coredns.io/plugins/kubernetes
discovery
Couchbase Autonomous couchbase.com/products/cloud/

Containerized Couchbase Open source
Operator kubernetes
Couchbase
Couchbase Server Self-managed NoSQL database By request couchbase.com/products/server
CRI-O CRI-O Lightweight container runtime Open source cri-o.io
Cloud-native control plane

Crossplane Crossplane Open source crossplane.io
framework
Crowdstrike
Falcon Humio Operator Kubernetes operator for Humio Open source github.com/humio/humio-operator
LogScale
Cloud-native distributed storage

CubeFS CubeFS Open source cubefs.io
platform
APIs for building portable

dapr dapr Open source dapr.io
microservices
Datadog Container datadoghq.com/product/container-

Datadog Container monitoring Trial period
Monitoring monitoring

Astra DB Multi-cloud DBaaS datastax.com/products/datastax-astra
datastax.com/products/astra-
Astra Streaming Multi-cloud streaming as a service
DataStax Trial period streaming
datastax.com/products/datastax-
DataStax Enterprise Open hybrid NoSQL database
enterprise
Containerized storage Kubernetes diamanti.com/products/ultima-

Ultima Enterprise
appliance enterprise
Diamanti By request
Multi-cloud-ready Kubernetes diamanti.com/products/ultima-
Ultima Accelerator
appliance accelerator
digitalocean.com/products/
DigitalOcean DigitalOcean Kubernetes Managed Kubernetes clusters By request
kubernetes
P2P-based image and file

Dragonfly Dragonfly Open source d7y.io
distribution system
Automatic observability for cloud-

Dynatrace Dynatrace Platform Trial period dynatrace.com
native workloads
Edge and service proxy for cloud-

Envoy Envoy Proxy Open source envoyproxy.io
native apps
etcd etcd Key value store for cluster data Open source etcd.io
Kubernetes security and

Fairwinds Fairwinds Insights Trial period fairwinds.com/insights
governance platform
Cloud-native runtime security and

Falco The Falco Project Open source falco.org
threat detection
docs.fluentd.org/v/0.12/articles/
Fluentd Kubernetes Fluentd Logging agent and data collector Open source
kubernetes-fluentd
CI/CD platform for deployment

GitHub Actions GitHub Actions Free tier github.com/actions
to Kubernetes
GitLab GitLab Cloud-based DevOps platform Free tier about.gitlab.com
Google Cloud Google Kubernetes Engine Managed Kubernetes platform Trial period cloud.google.com/kubernetes-engine
Cloud-native artifact registry and

Harbor Harbor Open source goharbor.io
container image management
Helm Helm Package manager for Kubernetes Open source helm.sh
IBM Cloud Kubernetes

IBM Managed Kubernetes platform Trial period ibm.com/cloud/kubernetes-service
Service
influxdata.com/solutions/
InfluxData InfluxDB Kubernetes monitoring solution Trial period
kubernetes-monitoring-solution
Istio Istio Service mesh for Kubernetes Open source istio.io
Operator for deploying and

Jaeger Jaeger Operator Open source jaegertracing.io
running Jaeger on Kubernetes
Kubernetes-based, event-driven
KEDA KEDA Open source keda.sh
autoscaler
Control plane for cloud-native app

Keptn Keptn Open source keptn.sh
DevOps automation
Serverless workload management

Knative Knative Open source knative.dev
on Kubernetes
Service mesh based on Kuma konghq.com/products/service-

Kong Kong Mesh By request
for Kubernetes mesh-platform
Kubernetes-native edge
KubeEdge KubeEdge Open source kubeedge.io
computing framework

Kubernetes add-on for virtualized

KubeVirt KubeVirt Open source kubevirt.io
workload management
Kuma Kuma Service mesh for Kubernetes Open source kuma.io
Kyverno Kyverno Native Kubernetes policy engine Open source kyverno.io
Linkerd Linkerd Service mesh for Kubernetes Open source linkerd.io
Linux Cloud-native CI/CD automation

Jenkins X Open source jenkins-x.io
Foundation solution
Microsoft azure.microsoft.com/en-us/products/
Azure Kubernetes Service Managed Kubernetes platform Trial period
Azure kubernetes-service
mirantis.com/software/mirantis-
Mirantis Kubernetes Engine Managed Kubernetes platform By request
kubernetes-engine
Container platform for Kubernetes mirantis.com/software/mirantis-

Mirantis Container Cloud Trial period
cluster deployment container-cloud
Mirantis
Container registry for Kubernetes mirantis.com/software/mirantis-
Mirantis Secure Registry
clusters secure-registry
By request
mirantis.com/software/container-
Mirantis Container Runtime Runtime for Kubernetes containers
runtime
Kubernetes network traffic nginx.com/products/nginx-ingress-

F5 NGINX Ingress Controller Trial period
management controller
NGINX
nginx.com/products/nginx-service-
F5 NGINX Service Mesh Service mesh for Kubernetes Open source
mesh
nutanix.com/products/kubernetes-
Nutanix Nutanix Kubernetes Engine Kubernetes management platform By request
engine
Ondat Kubernetes storage platform Free tier ondat.io
Kubernetes secrets management

Ondat Trousseau ondat.io/trousseau
Ondat framework
Open source
Kubernetes declarative disk
Ondat Discoblocks ondat.io/discoblocks
configuration system
Open Policy
Open Policy Agent Policy engine Open source openpolicyagent.org
Agent
OpenTelemetry OpenTelemetry Observability framework Open source opentelemetry.io
Operator
Operator Framework Kubernetes Operators framework Open source operatorframework.io
Framework
Oracle Cloud Infrastructure oracle.com/cloud/cloud-native/

Managed Docker registry service
Container Registry container-registry
Trial period
Oracle Cloud Container oracle.com/cloud/cloud-native/
Oracle Cloud Managed Kubernetes service
Engine for Kubernetes container-engine-kubernetes
Oracle Cloud Infrastructure Managed service mesh for oracle.com/cloud/cloud-native/

Free
Service Mesh Kubernetes service-mesh
Cloud-native app protection

Palo Alto Prisma Cloud By request paloaltonetworks.com/prisma/cloud
platform
github.com/planetscale/vitess-
PlanetScale Vitess Operator Kubernetes operator for Vitess Open source
operator
Platform9 Platform9 Managed Kubernetes platform By request platform9.com
Portainer Portainer Container management platform Free tier portainer.io

portworx.com/products/portworx-
Portworx Enterprise Kubernetes storage platform By request
enterprise
Kubernetes backup and data

Portworx Portworx Backup Open source portworx.com/products/px-backup
protection
portworx.com/products/portworx-
Portworx Data Services Kubernetes DBaaS platform By request
data-services
Cloud-native monitoring and

Prometheus Prometheus Open source prometheus.io
alerting platform
Rafay Rafay Kubernetes management platform Open source rafay.co
redhat.com/en/technologies/cloud-
OpenShift Kubernetes container platform
computing/openshift
Red Hat Trial period redhat.com/en/technologies/cloud-
OpenShift Container
PaaS, container platform computing/openshift/container-
Platform
platform
ReleaseHub ReleaseHub EaaS platform By request prod.releasehub.com
Kubernetes app delivery and

Replicated Replicated Trial period replicated.com
management platform
Rook Rook Storage orchestrator platform Open source rook.io
Istio-native API gateway and

Gloo Gateway solo.io/products/gloo-gateway
Kubernetes Ingress
Solo.io Trial period
Gloo Mesh Service mesh and control plane solo.io/products/gloo-mesh
SPIFFE SPIFFE Secure identity framework Open source spiffe.io
Platform and workload attestation

SPIFFE SPIRE Open source github.com/spiffe/spire
management
Ocean Container infrastructure engine spot.io/products/ocean

Spot By request
Cloud Analyzer Cloud cost management platform spot.io/products/cloud-analyzer
Sumo Logic Sumo Logic Continuous intelligence platform Trial period sumologic.com
Distributed block storage system

Longhorn longhorn.io
for Kubernetes
Rancher Multi-cluster orchestration platform rancher.com
SUSE K3s Kubernetes distribution for Edge/IoT Open source k3s.io
Cloud-native hyperconverged
Harvester harvesterhci.io
infrastructure
NeuVector Container security platform neuvector.com
Kubernetes, container, and cloud

Sysdig Falco Open source sysdig.com/opensource/falco
security
Sysdig
Kubernetes, container, and cloud
Sysdig Secure Trial period sysdig.com/products/secure
security
Thanos Thanos Prometheus support tool Open source thanos.io
Tigera Calico Cloud-native app protection platform Open source tigera.io/tigera-products/calico/
Ingress, API management, and

Traefik Labs Traefik Enterprise Trial period traefik.io/traefik-enterprise
service mesh
Cloud-native data protection and trilio.io/products/triliovault-

Trilio TrilioVault for Kubernetes Free tier
management platform kubernetes

Kubernetes-native API tyk.io/add-api-management-to-

Tyk Tyk Operator Free tier
management kubernetes
Database clustering system for

Vitess Vitess Open source vitess.io
horizontal scaling of MySQL
Container development,
Tanzu Build Service tanzu.vmware.com/build-service
management, and governance
By request
Tanzu Kubernetes Grid Kubernetes runtime tanzu.vmware.com/kubernetes-grid
VMWare
Tanzu Service Mesh Service mesh for Kubernetes tanzu.vmware.com/service-mesh
Multi-cloud Kubernetes operation

Tanzu Standard Edition Open source tanzu.vmware.com/tanzu/standard
and management
Cloud-native batch scheduling

Volcano Volcano Open source volcano.sh
system for Kubernetes
Weaveworks Weave GitOps Core Continuous delivery for Kubernetes Open source weave.works/product/gitops-core
Windocks Windocks SQL Server containers Free tier windocks.com

Dzone Kubernetes in The Enterprise 2022 1669219453241

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Dzone Kubernetes in The Enterprise 2022 1669219453241

Uploaded by

Copyright:

Available Formats

BROUGHT TO YOU IN PARTNERSHIP WITH

Even when building from a stable platform, it turns out that

Alan Hohn, Director of Software Strategy at Lockheed Martin

DZONE TREND REPORT | KUBERNETES IN THE ENTERPRISE PAGE 2

Key Research Findings

By Mike Gates, Guest Writer & Former Senior Editor at DZone

Major research targets included:

RESOURCE ISOLATION IN LINUX SYSTEMS

DZONE TREND REPORT | KUBERNETES IN THE ENTERPRISE PAGE 3

1. Higher-level containerization/orchestration tooling continues to mature, making them more attractive to

What methods do you use to isolate resources in Linux systems?

Results across all responses, comparing 2020–2022 data:

LINUX RESOURCE ISOLATION METHODS

% of Total Respondents Using

Method Used 2020 2021 2022

LXC 38.1% 32.8% 29.8%

chroot "jail" 31.2% 23.1% 27.4%

LXD 22.4% 21.6% 23.1%

LXCFS 18.8% 17.5% 18.4%

– First, as we hypothesized, container Table 2

(and container-related tooling) adoption

– For smaller companies, it simply might LXD 22.4% 21.6% 21.5%

DZONE TREND REPORT | KUBERNETES IN THE ENTERPRISE PAGE 4

LINUX RESOURCE ISOLATION METHODS BY ENVIRONMENT TYPE

% of Total Respondents Using

Development Environments Production Environments

Method Used 2020 2021 2022 2020 2021 2022

chroot "jail" 57.6% 61.6% 66.7% 42.4% 38.4% 33.3%

LXC 53.3% 56.3% 58.1% 46.7% 43.7% 41.9%

LXD 54.9% 62.4% 50.3% 45.1% 37.6% 49.7%

LXCFS 51.8% 55.3% 50.3% 48.2% 44.7% 49.7%

LINUX RESOURCE ISOLATION METHODS BY ENVIRONMENT TYPE AT LARGER COMPANIES

% of Total Respondents Using

Development Environments Production Environments

chroot "jail" 57.6% 61.6% 70.3% 42.4% 38.4% 29.7%

LXC 53.3% 56.3% 59.2% 46.7% 43.7% 40.8%

LXD 54.9% 62.4% 50.0% 45.1% 37.6% 50.0%

LXCFS 51.8% 55.3% 52.2% 48.2% 44.7% 47.8%

Results since 2017:

PERCENT OF RESPONDENTS USING CONTAINERS: 2017-2022

UNFILTERED DATA LARGER COMPANIES

DZONE TREND REPORT | KUBERNETES IN THE ENTERPRISE PAGE 5

Results (summed across 1,009 selections across all Table 5

categories [only displaying options that received > 5

OpenShift 28 5.0% 27 5.9%

Results since 2019:

SEE FIGURE 2 ON NEXT PAGE

DZONE TREND REPORT | KUBERNETES IN THE ENTERPRISE PAGE 6

PERCENT OF RESPONDENTS USING KUBERNETES

Results, year over year:

ATTITUDES TOWARD THE CURRENT STATE OF INFRASTRUCTURE ABSTRACTION

Opinion 2020 2021 2022

The cost in complexity of modern infrastructure abstraction is worth the

No opinion 13.7% 15.5% 7.6%

DZONE TREND REPORT | KUBERNETES IN THE ENTERPRISE PAGE 7

Research Target Two: Kubernetes — Promise vs. Performance

KUBERNETES: WHAT IS IT GOOD FOR?

What has Kubernetes improved at your organization?

had been previously stable. Reliability 46.0% 39.6% 48.2%

2. We were suspect of last year's drop in Cost 28.9% 28.2% 34.8%

DZONE TREND REPORT | KUBERNETES IN THE ENTERPRISE PAGE 8

THE STATE OF STATEFULNESS

Figure 1: The three pillars of observability 

WHY IS OBSERVABILITY IMPORTANT? 