Nine developer enablement

practices to achieve DevOps

at enterprise scale
CHRISTIAN OESTREICH
Director of Software Architecture &
Distinguished Engineer
About the Author
CHRISTIAN OESTREICH
Christian Oestreich has over 20 years of experience as a senior developer,
systems architect, and engineering team leader for Fortune 500
companies. He has expertise in leading large development teams in a
variety of software development methodologies, using a myriad of
different client and server technologies.
Table of Contents
03 The problem: A massively scaling dev team—but not enough ops talent

04 The situation today: A 25:1 dev-to-ops ratio

04 How we got to where we are today: The Metrics-Driven Mindset

06 One: Start with a performance requirements questionnaire

07 Two: Bake ops tasks into project bootstrapping

08 Three: Build libraries to accelerate code instrumentation

09 Four: Create patterns for custom metrics

10 Five: Integrate reporting agents into hosts and containers

11 Six: Automate platform compliance checks

12 Seven: Flow data into a single collection stream

13 Eight: Create a single source of truth for dashboards

13 Nine: Make software performance visible to all

14 Conclusion
The problem: A massively scaling
dev team—but not enough ops talent
In 2015, I found myself in a tough but familiar good ops engineers at the same rate that we
situation (and one that many readers will could find and hire developers.
recognize). Our team had a huge amount of
How were we going to support our rapidly
development to do on an ambitious new web
expanding development team? We needed to find
platform, and we were hiring developers by the
creative ways to empower our developers to write
dozen. But we couldn’t possibly scale the ops
high quality code, and then deploy, monitor, and
team to support the growth of the development
remediate problems themselves.
team. To put it simply, we couldn’t find and hire

03
NINE DE VELOPER ENAB LEMENT PR AC TICE S TO ACHIE VE DE VOP S AT ENTERPRI S E SCALE

The situation today: A 25:1 dev-to-ops ratio

In my 20 years of software development
experience, I’ve typically observed dev-to-ops
ratios of between 6:1 and 8:1. Fast-forward to
today: our organization’s dev-to-ops ratio is 25:1.
We have more than 500 engineers who have
delivered over 150 production microservices. Yet
only a small fraction of our engineers are focused
on architecture, operations, and governance. For
every 25 developers focused on feature delivery
and defect triage, there is only one engineer
dedicated to ops, architecture, and governance.

How we got to where we are today:

The Metrics-Driven Mindset
How did we achieve a 25:1 dev-to-ops ratio on a A well-planned metrics and monitoring strategy
team of more than 500 engineers? A big factor yields higher quality code, lower support costs,
was what I call our “metrics-driven mindset.” and more self-sufficient development teams.
I often pose this (rhetorical) question:
But, first, ops and architecture teams need
Why build software if you can’t (1) measure its to empower developers with the right tools
effectiveness and (2) react to issues in real-time? and processes to effectively monitor their
applications. In the pages that follow, I describe
To me, this is the core reason why developers
nine practices we’ve developed that were critical
(along with product managers and executives)
to our success:
need to constantly think about the key metrics
that measure their code’s success—and also why
ops and architecture teams need to make it easy
for developers to collect and report on those
metrics. Too often, metrics and monitoring are
an afterthought to software delivery when, in
reality, it needs to be a first-order consideration.

A well-planned metrics
and monitoring strategy
yields higher quality code,
lower support costs,
and more self-sufficient
development teams.

04
NINE DE VELOPER ENAB LEMENT PR AC TICE S TO ACHIE VE DE VOP S AT ENTERPRI S E SCALE

1. Start with a performance requirements questionnaire

2. Bake ops tasks into project bootstrapping

3. Build libraries to accelerate code instrumentation

4. Create patterns for custom metrics

5. Integrate reporting agents into hosts and containers

6. Automate platform compliance checks

7. Flow data into a single collection stream

8. Create a single source of truth for dashboards

9. Make software performance visible to all

These practices help ensure that developers are ops tasks. Below, I describe these principles in
focused on writing high-quality code that meets more detail and explain how we implemented
business requirements and not worrying about them in our environment.

05
NINE DE VELOPER ENAB LEMENT PR AC TICE S TO ACHIE VE DE VOP S AT ENTERPRI S E SCALE
01 Start with a performance requirements questionnaire
Almost immediately after the first container
starts, system and application metrics quickly
scale beyond the limits of human comprehension.
This leads to an overwhelming amount of data
and noise that can distract developers. So we
created a questionnaire that asks developers
Our performance requirements questionnaire
As part of the questionnaire, developers must
also think about alerting. I firmly believe that
if you’re going to monitor a metric, you should
also alert on it as well. A dashboard is fine, but
if something isn’t right, somebody should get a
to describe the top metrics they need to track
in order to ensure their code is functioning. We
encourage developers to work with business
partners to identify a handful of key indicators
per system.
notification. Therefore, this questionnaire also
forces developers and product teams to go
beyond metrics and think about key performance
indicators (KPIs) and service level indicators (SLIs).
106
NINE DE VELOPER ENAB LEMENT PR AC TICE S TO ACHIE VE DE VOP S AT ENTERPRI S E SCALE
As a sidenote, KPIs and SLIs illustrate contextual
critical business or system functions, whereas
metrics represent point-in-time data without
the necessary context. The key difference is in
defining what is “normal.” For example, the count
of records processed over the last 60 minutes is
a metric: it doesn’t provide any business context.
The successful percentage of records processed
over the last 60 minutes is a more useful indicator.
The count of records processed
over the last 60 minutes is a
metric: it doesn’t provide any
business context. The successful
percentage of records processed
over the last 60 minutes is a
more useful indicator.
02 Bake ops tasks into project bootstrapping
I’ve found that DevOps teams are often focused
more on automating code delivery pipelines
while pushing off post-production monitoring and
alerting to later. But these "later" stages of the
DevOps lifecycle are also important to automate.
It’s not efficient to think about monitoring and
alerting after the fact. On our team, monitoring
and alerting is baked into the earliest stages of
the software delivery lifecycle.
We’re a Java shop, specifically using the Spring
web framework and Spring Boot, an open
Ideally, developers fill out this questionnaire along
with their counterparts on the Product Team
before they begin writing code. The understanding
is that these metrics and alerts are going to be
visible to everybody (including executives). This
exercise should inform how developers write the
code. Then, when developers are getting ready
to deliver their code, the architecture team uses
this questionnaire during our governance check.
We verify (1) that the developers are going to
be accountable for the things they said were
important in the questionnaire; and (2) that
dashboards and alerts have been checked into
the repository.
source, Java-based framework used to create
microservices. We forked and customized the
Spring Initializr (a lightweight quickstart generator
for Spring projects). In our forked version of Spring
Initializr, we added additional automation for key
ops tasks. When developers click the microservice
generation button, it creates GitHub build streams,
metrics dashboards, and alert templates.
07
NINE DE VELOPER ENAB LEMENT PR AC TICE S TO ACHIE VE DE VOP S AT ENTERPRI S E SCALE

Our customized Microservice Generator automates code and ops tasks

So when developers want to bootstrap a project, are automated. We really want developers to be
they simply click through the Spring Initializr, as focused on writing high quality code, rather than
most of the manual microservice setup steps worrying about ops tasks.

03 Build libraries to accelerate code instrumentation

While it’s important to capture low-level application
runtime metrics, we don’t want developers spending
precious time creating that instrumentation.
Instead, we’d prefer that developers focus on the
business metrics that matter.
We built a library that all Java applications in
our stack use. If a developer is using the library,
application runtime metrics will be auto-
instrumented. The common Java library frees

08
NINE DE VELOPER ENAB LEMENT PR AC TICE S TO ACHIE VE DE VOP S AT ENTERPRI S E SCALE

developers to concentrate on instrumenting key and deploy this documentation to GitHub

business metrics, which are generally custom pages for the repository. This system provides
application metrics (in our case, things like health developers a valuable starting point and puts
data intake processing and success rate). the documentation at the source so people aren’t
scouring wikis for some obscure Word or
All repositories, regardless of business function,
Excel documents.
must contain documentation in the form of
AsciiDocs. This documentation is required
to provide context, sample usage, and SLA
requirements. The build pipeline will generate

Example documentation providing context on how to use our custom @SLA annotation

04 Create patterns for custom metrics

We aim to provide turnkey solutions which allow If the developers can instrument
developers to be naïve to the world of ops.
However, it is necessary for developers to instru-
that small number of KPIs and
ment any custom or application-specific metrics; SLIs, everything else should be
they just have to think about the specific key
delivered and built for them.
metrics they need to monitor for their applica-
tion. If the developers can instrument that small
number of KPIs and SLIs, everything else should
be delivered and built for them.

09
NINE DE VELOPER ENAB LEMENT PR AC TICE S TO ACHIE VE DE VOP S AT ENTERPRI S E SCALE
But first, developers need training and enablement.
We require all developers to complete a
microservice bootcamp. This bootcamp contains
all the coding patterns, including how to create
KPIs, SLIs, and custom metrics, that developers
need to get started when building services. This
bootcamp is “intra-sourced” and we encourage
active contributions, such as lessons learned and
new microservice patterns.
The bootcamp materials are currently written
in AsciiDocs, an excellent vehicle for linking and
embedding source files to provide additional
context and clarity. By directly embedding Java
05 Integrate reporting agents into hosts and containers
As developers get ready to deploy their code, it’s vital to ensure that the metrics will actually
get collected. To streamline the deployment process for developers, we automatically integrate
reporting agents into containers and hosts.
Hosts
Containers
files, the compilation and correctness of the
examples and solutions in the docs will have to
successfully compile and build to deploy new
documentation. This ensures that example code
will not be stale or broken.
During the bootcamp, we also introduce our
custom libraries and our AOP (aspect-oriented
programming) annotations, which experience
has shown us create the most useful and simple
high-level KPIs and SLIs. However, we do also
strongly encourage individual team creativity
when devising the metrics and visualizations
that will work best for their services.
Any host created inside our cloud environment
already has metric collection and aggregation
baked in at the OS layer.
All of our containers share a common base
container, which has an agent already baked in.
As a result, any new containers will inherit the
reporting agent. From there, we simply drop in
the application runtimes, and all of the monitoring
and logging is enabled.
10
NINE DE VELOPER ENAB LEMENT PR AC TICE S TO ACHIE VE DE VOP S AT ENTERPRI S E SCALE

06 Automate platform compliance checks

On our team, we’ve had a major push on not just High standards for platform compliance remove
software quality, but platform quality as well. many potential headaches for both dev and ops.

It's important to monitor what tools your devs are using before they deploy

With so many components and developers, it’s very
easy to have configuration and versioning drift.
To resolve this problem, we created an automated
compliance checker that checks the status of
roughly 20 framework dependencies, verifies the
build quality, and ensures that devs are using
up-to-date versions of the web framework, library,
containers, and other components. This final, pre-
deployment step helps developers avoid running
stale or insecure legacy items in production.
The compliance checker generates a compliance
score and a green or red result. If green, the
developer is clear to push to production. If red, the
developer understands that the microservice is no
longer compliant with our framework standards.
Updating everything to the latest versions is a click
of a button, as shown below.

11
NINE DE VELOPER ENAB LEMENT PR AC TICE S TO ACHIE VE DE VOP S AT ENTERPRI S E SCALE

The compliance checker interface makes it easy to see what’s out of compliance and make the
required updates.

07 Flow data into a single collection stream

Like most large enterprises, we have several
operations and monitoring platforms that we
need to unify. Having many systems throwing
out metrics and alerts is inefficient: it leads to
too many false positives and a lot of time wasted
switching between collection points and searching
for answers. Therefore, we ingest all streams into a
centralized platform which emits alerts.
In our unified data platform, teams can filter by
environment (either production, staging, or testing).
Teams can also decide which data is meaningful,
mute extraneous sources, and alert only on key
data. Plus, our ability to combine and correlate
monitoring data allows us to confidently diagnose
problems—and also provides additional context to
troubleshoot any issues that may arise.

12
NINE DE VELOPER ENAB LEMENT PR AC TICE S TO ACHIE VE DE VOP S AT ENTERPRI S E SCALE

08 Create a single source of truth for dashboards

With 500 developers empowered to create
dashboards and alerts, we needed to create
consistent standards of quality control. Too
many dashboards can be a problem: before you
know it, you could have 500 dashboards, though
few of them are meaningful. Rather than creating
many less-than-useful dashboards, we prefer
developers to create a limited number of
useful dashboards.
Rather than creating many
less-than-useful dashboards,
we prefer developers to
create a limited number
of useful dashboards.
To avoid dashboard overload, we export all
dashboards into a GitHub repository we call
“Datadog Ops,” which serves as the single source of
truth for dashboards. While individuals are able to
tinker with dashboards, every week we auto-deploy
the original dashboards back out of the repository
with the run of a Jenkins job and the Datadog API.
This mechanism keeps dashboards focused
and clean.
The Datadog Ops repository helps us cut down
on noise and maintain focus, while allowing
developers to experiment and create dashboards
on the fly. The repository also instantiates our
entire monitoring and alerting strategy as code,
and ensures that we have a way to automatically
recreate our entire dashboard and monitoring
infrastructure.
However, we do give developers the opportunity
to permanently modify their dashboards. We
periodically export “experimental” dashboards
and email their creators with this message:
“Please enter this dashboard into the Datadog
Ops repository if you’d like to replace the existing
production dashboard.”

09 Make software performance visible to all

We have large TVs around the office that
display our Datadog dashboards. We don’t put
low-level infrastructure or application runtime
metrics on these. Rather, these TVs are the
place for business-relevant KPIs and SLIs. The
high visibility makes everyone care a lot more
about monitoring their applications. In addition,
the high visibility of these dashboards creates
accountability for teams and encourages
collaboration among different teams (such as
business and technology) as well as various
levels (from front-line engineers to C-Suite).

13
NINE DE VELOPER ENAB LEMENT PR AC TICE S TO ACHIE VE DE VOP S AT ENTERPRI S E SCALE

A FEW ADDITIONAL TACTICAL TIPS:

·· Rotating screens (red/green or other complementary colors) are

I recommend limiting the number of displays easiest to see from a distance. Lines are great
(or rotating onscreen visualizations). We want to where variance is expected.
emphasize the few business-impacting metrics
·· Global & team-specific screens
that matter, so adding more data doesn’t
Include global business KPI boards on all TVs,
necessarily help.
but allow teams to rotate in team-specific
·· Information display boards on their TVs.
The less clutter on the screen the better;
generally large, color-coded numbers

Conclusion
The metrics-driven mindset has enabled the team
to deliver higher quality software faster. Our very
lean ops and architecture team has driven the
practices described above, freeing developers
to focus on writing good code and instrumenting
key business metrics, rather than worrying about
ops tasks. In addition, the metrics-driven mindset
gives devs ownership over their microservices
and encourages devs to identify and troubleshoot
issues themselves.

14