PART 2 DEVOPS Approach and Practices

The DevOps
Handbook

By: Gene Kim, Jez Humble,

Patrick Debois, and John Willis
 Outlines
▪ Introduction.
▪ Agile, Continuous Delivery,
▪ THE THREE WAYS
➢ The First Way: The Principles of Flow
➢ The Second Way: The Principles of Feedback
➢ The Third Way: The Principles of Continual Learning and Experimentation.
▪ WHERE TO START
▪ THE FIRST WAY: THE TECHNICAL PRACTICES OF FLOW.
▪ THE SECOND WAY: THE TECHNICAL PRACTICES OF Feedback
▪ THE THIRD WAY: THE TECHNICAL PRACTICES OF CONTIOUAL LEARNING.
▪ THE TECHNOLOGICAL PRACTICES OF INTEGRATING INFORMATION
SECURITY, CHANGE MANAGEMENT, AND COMPLIANCE.
THE THREE WAYS: THE PRINCIPLES
UNDERPINNING DEVOPS

Flow, System Thinking

Amplify Feedback Loops

Culture of Continual Experimentation & Learning

 The First way
The Principles of Flow

➢ Enables fast left-to-right flow of work from

Development to Operations to the customer.

➢ In order to maximize flow, we need to make work

visible, reduce our batch sizes and interval of work,
build in quality by preventing defects from being
passed to downstream work centers, and constantly
optimize for the global goals.
 The Principles of Flow
▪ MAKE WORK VISIBLE e.g. Sprint Board, KANBAN
Board
 The Principles of Flow

➢REDUCE Work In Progress WIP

❑ Bad multitasking: an engineer assigned to multiple
projects must switch between tasks, resulting in many
prioritization problems.

❑ We can limit multitasking when we use a kanban

board with WIP LIMITS.

❑ “Stop starting. Start finishing.” David J. Andersen,

author of Kanban.
 The Principles of Flow
➢REDUCE BATCH SIZES
❑ Mass production (Large Batch) vs Single piece (Small Batch) flow.
❑ Batch is code.
Simulation of “envelope game”
 The Principles of Flow
➢REDUCE THE NUMBER OF HANDOFFS
❑ Each time in Technology Value Stream, the work
passes from team to team, Potential queuing,
Waiting, Tickets, approvals, escalation.
❑ knowledge is inevitably lost with each handoff.
❑ Automate as much as possible, Reorganize the
team.
 The Principles of Flow
➢Continually identify and elevate our constraints
“In any value stream, there is always a direction of flow, and there is
always one and only constraint; any improvement not made at that
constraint is an illusion”.
“Five focusing steps:” ➢ Environment Creation.
❑ Identify the system’s constraint. ➢ Code Deployment.
❑ Decide how to exploit the constraint.
❑ Subordinate everything else to the above ➢ Test setup and run.
decisions. ➢ Overly tight architecture.
❑ Elevate the system’s constraint.
❑ If a constraint has been broken, go back
to step one, but do not allow inertia to
cause a system constraint.
 The Principles of Flow
➢ ELIMINATE HARDSHIPS AND WASTE IN THE VALUE
STREAM
❑ Partially done work.
❑ Extra processes, Extra Features.
❑ Task Switching, Waiting.
❑ Defects, Nonstandard or manual work.
❑ Heroics.
❑ Waiting.
❑ Motion.
❑ Defects.
❑ Nonstandard or manual work.
 The Second Way
The Principles of Feedback
➢ Enables the fast and constant flow of feedback
from right to left at all stages of our value stream. It
requires that we amplify feedback to prevent problems
from happening again, or enable faster detection and
recovery.

➢ By doing this, we create quality at the source and

generate or embed knowledge where it is needed -
this allows us to create ever-safer systems of work
where problems are found and fixed long before a
catastrophic failure occurs.
 The Principles of FEEDBACK
➢ WORKING SAFELY WITHIN COMPLEX SYSTEMS.
➢ SEE PROBLEMS AS THEY OCCUR.
➢ SWARM AND SOLVE PROBLEMS Immediately TO BUILD NEW
KNOWLEDGE.

➢ KEEP PUSHING QUALITY CLOSER TO THE SOURCE (Shift-t-Left,

Quality-In, Peer-Review, Check/Test Automation, Q/S is
everyone Responsibility).

➢ ENABLE OPTIMISING FOR DOWNSTREAM WORK CENTERS.

(Lean Internal Customer, Designing for Operation “ilities”)
The Principles of FEEDBACK
 The Principles of FEEDBACK

Outcomes Metrics:
• Deployment Pain/frequency.
• Change failure rate.
• Lead Time.
• Mean Time to Restore Service MTTR.
• Unplanned work.
• Employee Net Promoter Score (eNPS).
 The Third Way
The Principles of Continual Learning

➢ Enables the creation of generative,

high-trust culture that supports a
dynamic, disciplined, and scientific
approach to experimentation and risk-
taking, facilitating the creation of
organizational learning, both from
success and failures.
 The Principles of Continual
Learning
➢ ENABLE ORGANISATIONAL LEARNING AND A SAFETY CULTURE. (NO
“name, blame, and shame”, Instead “What caused the
problem?”).

➢ INSTITUTIONALISE THE IMPROVEMENT OF DAILY WORK. “Even

more important than daily work is the improvement of daily work.”

➢ TRANSFORM LOCAL DISCOVERIES INTO GLOBAL IMPROVEMENTS.

➢ INJECT RESILIENCE PATTERNS INTO OUR DAILY WORK. (reduce

deployment lead times, increase test coverage, decrease test
execution times, and even by re-architecting, game day exercises,
Chaos Monkey).

➢ LEADERS REINFORCE A LEARNING CULTURE.

 The Principles of Continual
Learning
The Westrum organizational typology model
WHERE TO START ?
 Green Field Brown Field
❖ Greenfield development is ❖ Brownfield development is
when we build on when we build on land that
undeveloped land. was previously used for
industrial purposes,
❖ Greenfield DevOps projects potentially contaminated with
are often pilots to hazardous waste or pollution.
demonstrate feasibility of ❖ Brownfield projects often
public or private clouds, come with significant
piloting deployment amounts of technical debt,
automation, and similar tools. such as having no test
automation or running on
unsupported platforms.
START WITH MOST SYMPATHETIC
AND INNOVATIVE GROUPS
➢CONSIDER BOTH SYSTEMS OF RECORD AND
SYSTEMS OF ENGAGEMENT.

➢EXPANDING DEVOPS ACROSS OUR

ORGANIZATION.
 IDENTIFY THE TEAMS SUPPORTING
OUR VALUE STREAM

• Product Owner.
• Development.
• QA.
• Operations.
• InfoSec.
• Release Manager.
• Technology Executives or Value Stream Manager.
CREATE A VALUE STREAM MAP TO
SEE THE WORK
 What Else?

▪ CREATE A DEDICATED TRANSFORMATION TEAM.

▪ AGREE ON A SHARED GOAL.
▪ KEEP OUR IMPROVEMENT PLANNING HORIZONS SHORT.
▪ RESERVE 20% OF CYCLES FOR NON-FUNCTIONAL
REQUIREMENTS AND REDUCING TECHNICAL DEBT.

▪ INCREASE THE VISIBILITY OF WORK.

▪ USE TOOLS TO REINFORCE DESIRED BEHAVIOR.
 CONWAY’S LAW
How to Design Our Organization and
Architecture with Conway’s Law in Mind

▪ “organizations which design systems... are

constrained to produce designs which are
copies of the communication structures of these
organizations…. The larger an organization is,
the less flexibility it has and the more
pronounced the phenomenon.”
 ORGANISATIONAL
ARCHETYPES

• Functional Oriented Organization

• Matrix Oriented Organization
• Market Oriented Organization

ENABLE MARKET-ORIENTED TEAMS (“OPTIMIZING

FOR SPEED”)
 Functional Matrix Market Oriented
Organization Organization Organization
• Optimize for expertise, • Mix of Both • Optimize for
division of labor, or responding quickly
reducing cost. to customer needs.
• Tall hierarchical • Flat organizational
organizational structure.
structure. • Adopt DevOps.
• Have Development &
Operations Team
separately.
❑PROBLEMS OFTEN CAUSED BY OVERLY FUNCTIONAL
ORIENTATION (“OPTIMIZING FOR COST”).
❑ENABLE MARKET-ORIENTED TEAMS (“OPTIMIZING FOR
SPEED”)
❑MAKING FUNCTIONAL ORIENTATION WORK.

❑TESTING, OPERATIONS, AND SECURITY AS EVERYONE’S

JOB, EVER.
❑ENABLE EVERY TEAM MEMBER TO BE A GENERALIST.

❑FUND NOT PROJECTS, BUT SERVICES AND PRODUCTS.

Functional vs Market Orientation
Specialists vs Generalists vs E-
shaped
 So, To get Greater Outcome
• Design Team boundaries in accordance with Conway’s Law.
• Create loosely coupled architectures to enable developer
productivity and safety.
• Keep team sizes small (The two pizza team rule).
• Create shared services to increase developer productivity.
• Embed DevOps engineer into our service teams.
• Assign an DevOps liaison to each service team.
• Integrate OPS into Dev Rituals.
• Invite DevOps to our dev stand-ups and retrospectives.
• Make relevant DevOps work visible on shared KANBAN board.
 The Technical
Practices of Flow
 The Technical Practices of
Flow
• Create the foundations of our deployment pipeline.
❑ Create single repository of truth for the entire system.
❑ Enable On-Demand creation of Dev, Test, and Production environments.
❖ Spinning up a new environment in Cloud e.g. AWS.
❖ Infrastructure as a Code e.g. Terraform.
❖ Configuration Management Tools e.g. Ansible.
❖ Virtualized environment e.g. Vagrant.

❑ Make Infrastructure easier to rebuild than to repair.

• Modify Definition of Development “DONE” to include running in

production-like environments.
 The Technical Practices of
Flow
• Enable Fast and Reliable Automated Testing.
• Continuously build, test, and integrate our code and
environments.
 The Technical Practices of
Flow
• Build a fast and reliable automated validation test suite

▪ Unit Tests. Single method, Function or Class tested in isolation.

‘Stub out’ Databases, Network calls

▪ Acceptance Tests. Test the application as a whole. Business

acceptance criteria of a User Story.

▪ Integration Tests. Testing application correctly interacts with

other production applications

• Catch errors as early in our automated testing as possible

The Technical Practices of
Flow
 The Technical Practices of
Flow
• Ensure tests run quickly(In parallel, if necessary).
• Test Driven Development. Write our Automated tests before we write the code.
❑ Ensure tests fail.

❑ Ensure tests pass.

❑ Refactor & Ensure tests pass.

• Automate as many of manual tests as possible.

• Integrate Performance Testing into our Test suite.
• Integrate Non functional requirements testing into our test suite.
❑ Availability, Scalability, Capacity, Security, and so forth.

• Pull ‘Andon Cord’ when the deployment pipeline breaks.

 The Technical Practices of
Flow

• Enable and practice CI.

• Adopt TRUNK based development practices.

 The Technical Practices of
Flow

• Automate and enable low risk releases.

• Ensure we maintain consistent environments.

• Deploying the same way to every environment.

• Automated Smoke testing our deployments.

 The Technical Practices of
Flow
❖Release Patterns
❑ Environment based release pattern
• Two or more environments. One environment receives customer’s live
traffic. New code is deployed to non-live environment, and the release
is performed moving traffic to this environment.

• E.g.

• Blue Green deployments pattern.

• Canary Releases.

• Cluster Immune Systems.

 The Technical Practices of
Flow
❖ Release Patterns
❑ Application based release pattern
• Modify the application and selectively release and expose specific
functionality by small configuration changes.

• E.g.

• Feature Flag. Visible to Development Team, Internal

employees, Selected customers.

• Dark Launch. Provides testing in production itself.

 The Technical Practices of
Flow
❖ Blue Green deployment pattern

❑ Create two databases

❑ Decouple database changes from application changes. Make only
additive changes and never mutate database objects.
 The Technical Practices of
Flow
• The Canary release pattern
• Cluster Immune System release patterns. Automatically
rollback based on performance threshold.
 The Technical Practices of
Flow
• Architect for Low-Risk Releases.

• Architectural Archetypes: Monoliths Vs Microservices.

• Use Strangler Application pattern to safely evolve our

enterprise architecture.
The Technical Practices of
Flow
 The Technical Practices of
Flow
Strangler Application Pattern

➢ Too tightly-coupled architecture.

➢ Develop, Test, and Deploy the decoupled functionality

independently.

➢ Placing existing functionality behind an API, where it

remains unchanged, and implementing new functionality
using our desired architecture, making calls to the old
system when necessary. When we implement strangler
applications, we seek to access all services through
versioned APIs, also called versioned services or
immutable services.
The Technical Practices
of Feedback
 The Technical Practices of
Feedback
CREATE TELEMETRY TO ENABLE SEEING AND SOLVING PROBLEMS
 The Technical Practices of
Feedback
• Create our centralized Telemetry Infrastructure.

❑ Data collection at the business logic, application, and

environments layer.

❑ An event router responsible for storing our events and metrics.

• Create application logging telemetry that helps solving production

issues.

❑ DEBUG, INFO, WARN, ERROR, FATAL.

 The Technical Practices of
Feedback
➢ Use Telemetry to Guide Problem Solving.
➢ Enable Creation of Production Metrics as Part of Daily work.
➢ Create self-service access to telemetry and information radiators.
➢ Find and fill any telemetry gaps.
• Business Level. e.g. Number of sales transactions, revenue of sales transactions, user
signups etc.
• Application Level. e.g. Transaction times, User response times, application faults etc.
• Infrastructure Level. e.g. WebServer traffic, CPU load, Disk usage etc.
• Client Software Level. e.g. JavaScript client errors, Mobile application errors, crashes
etc.
• Deployment Pipeline Level. e.g. deployment lead times, deployment frequencies,
environment status etc.
 The Technical Practices of
Feedback
• Analyze Telemetry to Better anticipate problems and
achieve goals. Use MEANS and STANDARD DEVIATIONS to
detect potential problems.

• ENABLE FEEDBACK SO DEVELOPMENT AND OPERATIONS CAN

SAFELY DEPLOY CODE.

• INTEGRATE HYPOTHESIS-DRIVEN DEVELOPMENT AND A/B

TESTING INTO OUR DAILY WORK.

• CREATE REVIEW AND COORDINATION PROCESSES TO INCREASE

QUALITY OF OUR CURRENT WORK.
 The Technical Practices of Continual
Learning & Experimentation

• ENABLE AND INJECT LEARNING INTO DAILY

WORK.
o Establish a Just, Learning Culture.
o Schedule Retrospective Meetings after Accidents Occur.
o Publish Our Retrospective Reviews as Widely as Possible.
o Decrease Incident Tolerances to Find Ever-Weaker Failure Signals.
o Redefine Failure and Encourage Calculated Risk-Taking.
o Inject Production Failures to Enable Resilience and Learning.
o Institute Game Days to Rehearse Failures.
 The Technical Practices of Continual
Learning & Experimentation

• CONVERT LOCAL DISCOVERIES INTO GLOBAL

IMPROVEMENTS.
o Use Chat Rooms and Chat Bots to Automate and Capture Organizational
Knowledge
o Hubot at GitHub.
o Automate Standardized Processes in Software for Reuse.
o Create a Single, Shared Source Code Repository for Our Entire Organization.
o Spread Knowledge by Using Automated Tests as Documentation and
Communities of Practice.
o Design for Operations through Codified Non-Functional Requirements.
o Build Reusable Operations User Stories into Development.
o Ensure Technology Choices Help Achieve Organizational Goals.
 The Technical Practices of Continual
Learning & Experimentation

• RESERVE TIME TO CREATE ORGANIZATIONAL

LEARNING AND IMPROVEMENT
o Institutionalize Rituals to Pay Down Technical Debt.
o Enable Everyone to Teach and Learn.
o Share your Experiences from DevOps Conferences.
o Create Community Structures to Spread Practices.
 The Technical Practices of Integrating
Information Security, Change
Management, And Compliance
• INFORMATION SECURITY IS EVERYONE’S JOB
EVERY DAY
o Integrate Security into Development Iteration demonstration.
o Integrate Security into Defect Tracking and Post-Mortems.
o Integrate Preventive Security Controls into Shared Source Code
Repositories and Shared Services.
o Integrate Security into Our Deployment Pipeline.
o Ensure Security of the Application.
o Ensure Security of Our Software Supply Chain.
o Ensure Security of the Environment.
o Integrate Information Security into Production Telemetry.
o Protect Our Deployment Pipeline.
 The Technical Practices of Integrating
Information Security, Change
Management, And Compliance

• PROTECTING THE DEPLOYMENT PIPELINE

o Integrate Security and Compliance into Change Approval
Processes.
o Recategorize Lower-Risk Changes as Standard Changes.
o What to Do When Changes Are Categorized as Normal
Changes.
o Implement Separation of Duty through Code Review.
o Ensure Documentation and Proof for Auditors and Compliance
Officers.
Gene Kim,

Thank You