Agile DevOps: The flattening of the software release

process
How DevOps is transforming software delivery

Paul Duvall (paul.duvall@stelligent.com) 23 August 2012

CTO
Stelligent

What does it mean to "flatten" your software release process? How will this affect your
organizational structure? In the first installment of the Agile DevOps series, DevOps expert Paul
Duvall describes how developers and operations are working together on software delivery
teams to streamline the process of developing and releasing software. He discusses such
nascent topics as test-driven infrastructures, transient environments, and the Chaos Monkey
and how these techniques all work toward the goal of getting software to users more quickly
and more often.

View more content in this series

About this series

Developers can learn a lot from operations, and operations can learn a lot from developers.
This series of articles is dedicated to exploring the practical uses of applying an operations
mindset to development, and vice versa and of considering software products as holistic
entities that can be delivered with more agility and frequency than ever before.

When it comes to delivering software to users, Agile development has yet to live up to some of
its promises in many organizations. The reasons for this are manyfold, but the central problem is
a misalignment in organizational structure. More often than not, a company's development and
operations teams are in separate corporate units with only one executive in common the CEO,
who usually trusts the subordinate executives to make appropriate decisions. The two teams' aims
are in inherent conflict. Development is measured by the number of features it makes available to
the business, while operations is measured by the system's up time its stability. Because such
companies lack a cohesive view of the software system, neither team is properly incentivized to
achieve the goal of regularly delivering high-quality features that run in a stable environment.

Thomas Friedman's 2005 book The World is Flat makes the case that factors such as
globalization, software, the Internet, and the opening of certain developing countries' borders are

Copyright IBM Corporation 2012 Trademarks

Agile DevOps: The flattening of the software release process Page 1 of 9
developerWorks ibm.com/developerWorks/

converging to "flatten" traditional barriers to participation in the world's economy. The software
industry is now seeing its own "flattening" trend within companies that seek competitive advantage:
the flattening of software releases and organizational structures. This transformation made
possible by a convergence of automation, tools, collaboration, and industry best practices and
patterns is making software delivery at these companies faster, better, and cheaper.

In this series, I'll show how software systems can be viewed more holistically and how software
organizations must be flattened to achieve the objective of regular delivery of stable features.
This article introduces many of the topics that I'll cover throughout the series, including Agile,
DevOps, emerging industry patterns, tools, and collaboration. All of them contribute to achieving
this objective.

Get involved
developerWorks Agile transformation provides news, discussions, and training to help you
and your organization build a foundation on agile development principles.

Agile + DevOps
Three of the 12 principles in the Agile Manifesto (see Resources) emphasize software delivery
practices:

"Our highest priority is to satisfy the customer through early and continuous delivery of
valuable software."
"Deliver working software frequently, from a couple of weeks to a couple of months, with a
preference to the shorter timescale."
"Working software is the primary measure of progress."
The Manifesto was written more than 10 years ago, and many organizations are still seeking to
adhere to these principles. The exciting thing, though, is that other organizations are exceeding
these tenets. In some companies, the software is always ready to release, and there are no walls
or silos around the teams: they have one delivery team consisting of experts in development,
QA, database, operations, and so on continually delivering valuable software to their users. This
is what agility is all about.

At its heart, the motivation leading to the DevOps movement is the frustration involved in releasing
and maintaining software systems across teams. On the typical software project, and broadly
speaking, two teams are responsible for developing and delivering software: development and
operations. The two teams must collaborate, but they often have competing motivations. For
development, it's delivering new features. For operations, it's ensuring the stability of the systems.
Often, there's miscommunication between teams, which leads to failures late in the software
delivery process.

One of the goals of the DevOps movement is for developers and operations people to work
together in a more collaborative fashion. As a result, a new breed of DevOps engineers is
emerging who take the best of both disciplines and combine them to deliver value to users. This
is also manifesting in the rise of cross-functional teams that are experienced in development,
configuration management, database, testing, and infrastructure.

Agile DevOps: The flattening of the software release process Page 2 of 9

ibm.com/developerWorks/ developerWorks

Patterns
A software pattern is a solution to a problem in a particular context (context being a key to
distinguishing patterns from "best practices"). Some of the patterns related to Agile DevOps are
scripted environments, test-driven infrastructures, Chaos Monkey, versioning everything, delivery
pipeline, and DevOps dashboard. I introduce these patterns in this article, and you'll see more
detail about them as the series progresses.

Scripted environments
By fully automating environment provisioning, you reduce the risks of deployment errors occurring
in one environment that do not occur in other environments. By scripting environments, you verify
the integrity of a particular version of the software (that is, that nothing has been modified) in target
environments. By following the policy that nothing gets modified to an environment unless it's in a
versioned script, automated, and part of a single path to production, you can better determine the
root cause of deployment errors. The benefits of scripted environments include:

Environments are always in a known state.

They enable self-service environments and deployments.
They lessen the chance that deployments behave differently based on unique environment
types.
Environments are a part of a single path (and version) to production.
They lessen the chance that knowledge is locked in team members' heads.
Deployments are more predictable and repeatable.
Infrastructure automation tools that you'll learn more about in this series support this pattern.
Puppet is one such tool. The code snippet in Listing 1 exemplifies a basic Puppet program
known as a manifest:

Listing 1. Scripted environments: Puppet manifest httpd snippet

class httpd {
package { 'httpd-devel':
ensure => installed,
}
service { 'httpd':
ensure => running,
enable => true,
subscribe => Package['httpd-devel'],
}
}

Typically, numerous manifests script an infrastructure. These scripts are committed to a version-
control repository just like application code.

Test-driven infrastructures
A simple premise of DevOps is the application of the best practices and patterns from
development to operations and from operations to development. The test-driven approach
of writing tests along with the code has its origins in software development. As infrastructure
automation tools become more mainstream across organizations, engineers are applying test-

Agile DevOps: The flattening of the software release process Page 3 of 9

developerWorks ibm.com/developerWorks/

driven practices to their infrastructure; the infrastructure is described in a script (as you saw in
Listing 1), and these scripts have tests. Some of the many benefits of test-driven infrastructures
are:

Problems manifest earlier because infrastructure changes are integrated with the rest of the
software system through a Continuous Integration system.
The tests and the scripted infrastructure become documentation.
You can better isolate destructive changes, because everything is a script and described in
tests.

Listing 2 illustrates a simple scenario of verifying that a web server is up and running. It uses a
behavior-driven development (also known as acceptance-test driven development) tool called
Cucumber, in which tests are described as scenarios.

Listing 2. Infrastructure test in Cucumber

Scenario: Is the proper version of Apache installed?
When I run "/usr/sbin/httpd -v"
Then I should see "2.2"

You'll see more test-driven infrastructure examples in future articles in this series.

Chaos Monkey
A couple of years ago, the Netflix tech team developed a continuous testing tool it calls the Chaos
Monkey. The tool intentionally and randomly, but regularly, terminates instances in the Netflix
production infrastructure to ensure that the systems continue to operate in the event of failure. This
tool was recently released as open source (see Resources).

The principle Netflix follows is that "everything fails, all the time" (a mantra coined by Werner
Vogels, CTO of Amazon.com). As Netflix states, "The Chaos Monkey's job is to randomly kill
instances and services within our architecture. If we aren't constantly testing our ability to succeed
despite failure, then it isn't likely to work when it matters most in the event of an unexpected
outage" (see Resources). Later on in this series, I'll explore the Chaos Monkey and its role in a
well-functioning DevOps environment.

Transient environments
The general rule of thumb with a transient environment is that self-service environments delivered
to DevOps team members are as short-lived as possible, anywhere from a few hours to a few
days basically, just enough time to run through tests. One of the more significant problems
on software-development projects is teams having fixed instances that no one else can touch
because they took various team members days, weeks, or months to configure. This often is
the result of environments that aren't scripted, making environments scarce resources. When
something is scarce, you want to hold on to it and protect it.

When you have fully scripted environments, environments are no longer scarce. Everything
is defined in a template and checked into a version-control system. You can terminate the

Agile DevOps: The flattening of the software release process Page 4 of 9

ibm.com/developerWorks/ developerWorks

environments based on the scripts as often as necessary. This pattern offers two advantages:
it makes the resources available for someone else to use, and it reinforces the concept that
everything must be automated not nurtured and cultivated manually over weeks and months.
You'll learn more about transient environments in this series.

Version everything
This pattern is simple and seemingly self-evident: version everything. Yes, everything:
infrastructure, configuration, application code, and database scripts. Although that may be simple
to understand, in my experience it's rare to find teams that version all of the artifacts required to
create the software system.

The purpose of versioning everything is to establish a single source of truth (also known as
canonical version or the system of record) for the software system; you treat software as a holistic
unit. When versioning everything, team members aren't continually unclear or navigating myriad
versions of the software system. In the series, I'll also cover an emerging practice of versioning
parts of the software delivery system configuration itself.

There's one simple way to know if everything is being versioned: a new person on the team who
gets a new machine and performs a single-command checkout from version control should be able
to get a complete working software system from that checkout.

Delivery pipeline
With a Continuous Integration server such as Jenkins, you can also configure a delivery pipeline.
Essentially, a delivery pipeline is a process in which different types of jobs are run based on the
success of the previous job. In this pipeline, you can configure various stages including a commit
stage, an acceptance stage, and so on. Each stage builds on the success of the previous stage;
this approach ensures that you're reducing the risk of the release candidate in each successive
stage until the software system is released to production. An example of a delivery pipeline as
represented in Jenkins is shown in Figure 1:

Figure 1. Delivery pipeline as represented in the Jenkins continuous-

integration server

DevOps dashboard
Once you have a cross-functional delivery team focused on delivering features and stability, it
becomes immensely useful to get everyone on the same page both figuratively and literally.

Agile DevOps: The flattening of the software release process Page 5 of 9

developerWorks ibm.com/developerWorks/

A DevOps dashboard shows how each change affects the entire system in every stage in the
progression to deploying to production. Later in the series, I'll detail an open source dashboard that
provides real-time information on the software system under development.

Collaboration
Collaboration is one of the pillars of DevOps. Traditional development and operations teams tend
to work in silos and to limit the amount of interteam communication until software release times.
Ensuring collective ownership, establishing cross-functional teams, and broadening the skill sets
of engineers are three ways to increase collaboration and break down the traditional barriers that
prevent software from being delivered regularly.

Collective ownership
Collective ownership is a practice dating back to the establishment of extreme programming (XP)
and, generally speaking, Agile methodologies. In the context of continuous delivery, however, the
emphasis is on ensuring that all types of source files that make up a software system are available
for any authorized team member to modify. These include the application code, configuration, the
data, and even the infrastructure. Everything is scripted, and everything can be modified.

Cross-functional teams
Having a cross-functional team means that every team member is responsible for the delivery
process. Any person on the team can modify any part of the software system. The corresponding
antipattern is siloed teams: Development, testing, and operations have their own scripts and
processes and are not part of the same team.

Cross-functional teams consist of representatives from all the necessary disciplines. Instead
of treating each discipline as a separate centralized service organization, the delivery team
becomes the key organizational construct. Teams work together in a dedicated fashion to deliver
software consistently, without the time impediments inherent when teams communicate across
the organization. Consider composing each team of (at least) business analysts, customer
representatives, DBAs, developers, project managers, and QA and release engineers. With a
cross-functional team, you reduce the "it's not my job" syndrome that stifles communication and
the numerous "walls" preventing effective communication that are established between teams
within and across physical locations.

Polyskilled engineers
A polyskilled engineer is a team member who is skilled in every area of the software delivery
process. Developers should understand how to make database changes. Database administrators
should understand how to write functional tests. Project managers should understand how to
add scenarios to automated tests. In general, team members continue to perform the functions
consistent with their roles. (For example, DBAs still generally focus on databases.) However, the
shared knowledge greatly reduces the communication barriers introduced in large, geographically
dispersed organizations. It also limits the need to rely on a few key individuals to get software
released to users.

Agile DevOps: The flattening of the software release process Page 6 of 9

ibm.com/developerWorks/ developerWorks

Tools
Tools are an important component of Agile DevOps. The tools you choose for your project depend
on your particular requirements, but one essential specification is that the tool must run from
the command line. This is because you want the pipeline to run in headless mode or with just
one click. (If you have a legacy tool that does not provide a command-line option, you can try
screen scrape to run in headless mode, but this is less than ideal and often requires regular
maintenance.) In Table 1, you see a list of some of the types of tools in a typical DevOps toolset;
focus more on the types of the tools rather than the tool names:

Table 1. Tools that support DevOps work

Tool Type Tools

Infrastructure automation Bcfg2, CFEngine, Chef, CloudFormation, IBM Tivoli, Puppet

Deployment automation Capistrano, ControlTier, Func, Glu, RunDeck

Infrastructure as a Service Amazon Web Services, CloudStack, IBM SmartCloud, OpenStack,

Rackspace

Build automation Ant, Maven, Rake, Gradle

Test automation JUnit, Selenium, Cucumber, easyb

Version control Subversion, Git, IBM Rational ClearCase

Continuous Integration Jenkins, CruiseControl, IBM Rational BuildForge

Keep in mind that the list in Table 1 is meant to be illustrative, not exhaustive (see Resources for a
more detailed list of tools).

Smoother roads ahead for software delivery

The flattening of software releases and the organizations is enabled through Agile DevOps. In this
article, I've commented on the state of Agile, what DevOps is, and how developers and operations
can work more collaboratively to deliver features in stable environments more often. I introduced
some of the emerging DevOps patterns that I will cover in more detail in future articles in this
series. Finally, I provided a list of some of the tools that support Agile DevOps software delivery.

In the next article, you'll learn about the infrastructure automation tools that support the scripted
environments pattern. I'll cover the leading open source infrastructure automation tools: Chef and
Puppet.

Agile DevOps: The flattening of the software release process Page 7 of 9

developerWorks ibm.com/developerWorks/

Resources
Learn

Principles behind the Agile Manifesto (Agile Manifesto, 2001): The Manifesto's principles
stress frequent delivery of working software.
DevOps: Wikipedia describes the methodologies and motivation behind the DevOps
movement.
"5 Lessons We've Learned Using AWS" (John Ciancutti, The Netflix Tech Blog, December
2010): The Chaos Monkey was one of the first tools that Netflix engineers built to support its
cloud systems.
Continuous Delivery Tools List: Stelligent provides an exhaustive list of tools for implementing
continuous delivery.
"DevOps for mobile development" (Michael Rowe, developerWorks, July 2012): Find out
how DevOps can help address the issues of deploying different versions of apps to different
devices.
Stay current with developerWorks technical events and webcasts focused on a variety of
IBM products and IT industry topics.
Attend a free developerWorks Live! briefing to get up-to-speed quickly on IBM products and
tools as well as IT industry trends.
Follow developerWorks on Twitter.
Watch developerWorks on-demand demos ranging from product installation and setup demos
for beginners, to advanced functionality for experienced developers.

Get products and technologies

Chaos Monkey: Chaos Monkey is the first software release in the Netflix "Simian Army."
Evaluate IBM products in the way that suits you best: Download a product trial, try a product
online, use a product in a cloud environment, or spend a few hours in the SOA Sandbox
learning how to implement Service Oriented Architecture efficiently.

Discuss

Get involved in the developerWorks community. Connect with other developerWorks users
while exploring the developer-driven blogs, forums, groups, and wikis.
The developerWorks Agile Transformation community provides news, discussions, and
training to help you and your organization build a foundation on agile development principles.

Agile DevOps: The flattening of the software release process Page 8 of 9

ibm.com/developerWorks/ developerWorks

About the author

Paul Duvall

Paul Duvall is the CTO of Stelligent. A featured speaker at many leading software
conferences, he has worked in virtually every role on software projects: developer,
project manager, architect, and tester. He is the principal author of Continuous
Integration: Improving Software Quality and Reducing Risk (Addison-Wesley, 2007)
and a 2008 Jolt Award Winner. He is also the author of Startup@Cloud and DevOps
in the Cloud LiveLessons (Pearson Education, June 2012). He's contributed to
several other books as well. Paul authored the 20-article Automation for the people
series on developerWorks. He is passionate about getting high-quality software to
users quicker and more often through continuous delivery and the cloud. Read his
blog at Stelligent.com.

Copyright IBM Corporation 2012

(www.ibm.com/legal/copytrade.shtml)
Trademarks
(www.ibm.com/developerworks/ibm/trademarks/)

Agile DevOps: The flattening of the software release process Page 9 of 9