Metrics to Power DevOps

Toward making decisions based on data – not opinion

Kevin Hancock Sujoy Sen

Sr. Director, Field Engineering Vice President

CollabNet, Inc. Practice Head – IT Transformation Consulting

1 Copyright ©2014 CollabNet, Inc. All Rights Reserved.

 Today’s Presenters
Kevin Hancock, Senior Director Worldwide Field Operations
In his ten plus year at CollabNet, Kevin has led successful implementations of
CollabNet products and services at some of the world’s largest enterprises.
Kevin works with organizations to transform their globally distributed application
development teams into modern, agile organizations, helping to save millions of
dollars while accelerating software delivery. Kevin holds a Bachelor of Science
degree from Carnegie Mellon University, and a Masters degree from the
University of Pittsburgh.

Sujoy Sen, Practice Head – IT Transformation Consulting

Sujoy is part of the leadership team in the Tech Advisory group of TechMahindra.
He is heading a practice which helps customers improve the success rate of their
critical IT projects. Sujoy is a Certified Six Sigma Black Belt & Manager of
Organizational Excellence from American Society for Quality, a PMP, a CISA and
an Agile Trainer & Coach.
Sujoy has more than 19 years of experience across various domains and
functions. He has been part of Tech Mahindra’s IT & Quality consulting group for
the last 10 years.

2 Copyright ©2014 CollabNet, Inc. All Rights Reserved.

 Agenda

• Where are we now?

• What’s the problem?
• What do we need?
• How do we get there?

3 Copyright ©2014 CollabNet, Inc. All Rights Reserved.

 Where are we now?

4 Copyright ©2014 CollabNet, Inc. All Rights Reserved.

IT Priorities are changing
The focus has shifted from reducing cost to improving
the effectiveness of business processes
• IT is expected to improve business process effectiveness
• IT spending on innovation is expected to increase by 50% and
spending on transaction activities is expected to go down by
15%
• Further current spending on next gen infrastructure is at it
highest in 5 years
5 Copyright ©2014 CollabNet, Inc. All Rights Reserved.
McKinsey Global Survey, 2014
• To address the concerns and make effectives use of additional
infrastructure spending, IT should be more agile and responsive
resulting in more innovation
• To enable innovation, a culture of experimentation where IT is
allowed to test new innovations quickly with customer, capture
feedback and continuously improve
• This requires the ability of continuous delivery resulting in better,
faster and cheaper innovation
 Agile Upstream
Agile
• Agile Upstream are those
upstream practices usually associated
with planning, managing and
executing a Software project.
• Scrum, Kanban, XP are all
used upstream in the
32%* process.
• But what about
Downstream?

*Do standup, burndown/burnup, and

short iterations; product owner; and
constant feedback.
Source:
6 February 5, 2014, “How Can You Scale Your©2014
Copyright AgileCollabNet,
Adoption?”Inc. All Forrester report
Rights Reserved.
 Agile
• Agile Downstream refers to Test
automation (Test-driven downstream
Development). Continuous
Integration, Continuous
Delivery.
• Test automation is a key part of
any Continuous Integration 13%†
process
• Successful Continuous
Integration requires Build
Management
• Successful Continuous Delivery †Do upstream plus continuous
requires Release Management integration, continuous delivery, and
test-driven development.
Source:
7 February 5, 2014, “How Can You Scale Your©2014
Copyright AgileCollabNet,
Adoption?”Inc. All Forrester report
Rights Reserved.
 We need an Agile System
• Agile Software
Development
• Continuous
Integration
• Continuous Delivery
• Visible and accessible
Metrics
• Access for ALL
Stakeholders (RBAC
permitting)

8 Copyright ©2014 CollabNet, Inc. All Rights Reserved.

 Few Companies Have Achieved Enterprise Agility

<20% A few %
Enterprise

Have Scaled Agile Have achieved

across their organization Enterprise Agility

32% 13%
Workgroup

Perform Upstream Agile: Perform Upstream + Downstream

standup, burndown/burnup, Agile: Perform upstream plus
and short iterations; product continuous integration, continuous
owner; and constant delivery, and test-driven development
feedback

Plan (Upstream) Deploy (Downstream)

9 Copyright ©2014 CollabNet, Inc. All Rights Reserved.

 What’s the Problem?

10 Copyright ©2014 CollabNet, Inc. All Rights Reserved.

 Silos of Data and Automation

• Highly optimized silos or work centers

make sense for Waterfall
– Trust each work center to build the most
“perfect product” to pass to the next work
center
– We know this rarely works
• Agile Enterprises optimize for the
System
– Identify and fix bottlenecks
– Only improve a work center if it improves the
system

11 Copyright ©2014 CollabNet, Inc. All Rights Reserved.

 A model for DevOps Empowerment

• The first way:

– Systems Thinking

• The second way:

– Amplify feedback loops

• The third way:

– Culture of continual experimentation and
learning

12 Copyright ©2014 CollabNet, Inc. All Rights Reserved.

The First Way: Systems Thinking

• Focus on value streams

• Think globally
• Understand the system as a whole
• Stop problems early and often

• Centralised source code and

environment configuration.
• Consistent and automated
release processes.
Development Operations
• Decreased cycle times,
(business) (customer)
faster release cadence.

13 Copyright ©2014 CollabNet, Inc. All Rights Reserved.

 The second way: Amplify Feedback Loops
• Protect the integrity of the entire system of work, versus
completion of tasks
• Expose visual data so everyone can see how their decisions
affect the entire system

• Find and fix defects,

performance and availability
issues faster.
• Enable specialised functions
Development Operations
such as Ops and InfoSec to
(business) collaborate more effectively. (customer)
• Increase communication across
the entire business.
• Get more work done

14 Copyright ©2014 CollabNet, Inc. All Rights Reserved.

 The Third way: Culture of continual experimentation and learning
• Continual experimentation
• Tasking risks and learning from failure
• Repetition and practice - pre-requisites to mastery
• Create rituals that reward risk taking
• Introduce faults to increase resilience

Development Operations
(business) (customer)

15 Copyright ©2014 CollabNet, Inc. All Rights Reserved.

 What Do We Need?

16 Copyright ©2014 CollabNet, Inc. All Rights Reserved.

 Key Metrics that Power DevOps
1. Change Volume
2. Lead Time (from dev to deploy)
3. Deployment Frequency
4. Percentage of Failed Deployments
5. Build Count By Day
6. File Churn
7. Build duration
8. Cycle time

Open ALM
Platform

Enterprise DevOps Metrics

require consistency across
Traditional agile metrics
projects, enterprise-wide
have focused here
visibility, and the ability to drill
to project data and people

17 Copyright ©2014 CollabNet, Inc. All Rights Reserved.

 Portfolio Dashboards
• As an executive, I need to see across projects to determine the
relative success of different teams.

Any metric is more valuable when it is put into context, and

compared with other teams.

18 Copyright ©2014 CollabNet, Inc. All Rights Reserved.

 Change Volume
• How many user stories and new lines of code are being
deployed?

The key is to understand the risk of a given deployment – more user

stories and more lines of code adds risk.

19 Copyright ©2014 CollabNet, Inc. All Rights Reserved.

 Average Age (from development to deployment)
• How long does it take for a new requirement/story/task tp go
from the time when it hits development to when it successfully
gets deployed into production.

Average age is an important indicator of efficiency in the process – when tracked using
value stream mapping, it can help the team to visualize areas in the process which need
improvement. Cycle time should reduce as the team gets a better hold of the lifecycle.
20 Copyright ©2014 CollabNet, Inc. All Rights Reserved.
 Team Deployments
• How often is the team deploying new code?

For an Agile team delivering code in 2 or 4-week sprints, this metric should
become predictable, and correlate closely, if not exactly to those sprints.

21 Copyright ©2014 CollabNet, Inc. All Rights Reserved.

 Build Duration
• On average, how long are builds taking to complete?

This metrics should remain stable over time. Spikes in build duration should be
red-flagged and investigated.

22 Copyright ©2014 CollabNet, Inc. All Rights Reserved.

 Cycle Time
• How is the team trending in terms of cycle time, over the
lifecycle of a project?

Technical debt will add to cycle time, but so will general complexity. Most
projects start out strong, but over time degrade. Choose the level that is
acceptable and seek stability at that level.
23 Copyright ©2014 CollabNet, Inc. All Rights Reserved.
 Velocity and Estimation
• How are teams doing? Will they finish what is promised?

Velocity will let us know how much work a team can get done, if they are
consistent in their estimation and improving; while the burnup will let us
know if we are finishing work quicker than it is being added to the queue

24 Copyright ©2014 CollabNet, Inc. All Rights Reserved.

 How Do We Get There?
The Cloud Framework to Power DevOps

25 Copyright ©2014 CollabNet, Inc. All Rights Reserved.

 Connect Teams, Processes, and Tools

26 Copyright ©2014 CollabNet, Inc. All Rights Reserved.

 TechM ADOPT Platform with TeamForge

Dashboards
GitEye

Control center for delivery pipelines, with social activity streams and open pluggable API’s

Tracker Discussions Wiki Documents Reports File Releases

Git + Gerrit Hudson

JIRA Code Search Jenkins Your data center

Code Review Selenium

PPM
QTP
Public clouds

Platform 4 One Platform, Multiple Possibilities

27 Copyright ©2014 CollabNet, Inc. All Rights Reserved.

 Tool Snapshot

Unit Test / Code Traceability for

Configuration Customized
Agile Backlogs quality tool application lifecycle
Project Management Reports
integration activities
Dashboard

Code Review Infrastructure

Jenkins Collaboration Tool Automation
Task integration Tools
Tracking /
Managing
Sprint Defect
Tracker

Defect
Tracker/Impediment
Release Tracker
Planning

Platform 4

28 Copyright ©2014 CollabNet, Inc. All Rights Reserved.

 Focus on Cadence and Orchestration

Sprint
Dev (D) Build (B) Test (T) QA Prod (P)
Continuous Integration (Q)
• Devops should create a
Continuous Testing flow that results in faster
Daily/Release through put time
Daily Daily
point
• Our approach ensures that
Continuous Planning

Unit tests
we improve the overall
Scrum
Team 1 Build? Automate throughput by bringing in
d Tests
process, tools, automation
Unit tests
and training interventions
Build? D T A Prod
where the flow gets stuck
Unit tests

Scrum Build?
Automate • TechM proposes a holistic
d Tests
Team 2 approach to enable Devops
in an organization by
Unit tests
slowing eliminating legacy
Continuous Deployment practices which don’t
promote collaboration
Continuous Release & Monitoring

Continuous Delivery

29 Copyright ©2014 CollabNet, Inc. All Rights Reserved.

 Simplified process for continuous delivery
1
Processes
1
Processes S Start E End

Business Products Product Product Epics (product

S
Idea vision Identification Roadmap backlog)
I2T to
task Sprint Plans
Tasks Release Plans
(Sprint backlog)

Automated Automated Code Check-in to

Design Code
Task Unit test quality check repository
T2P to
Package Package
Automated Trigger
Build Integration

Deploy to Manual/ Automated Deploy to Manual / Automated UAT

QA Env* SIT Testing UAT Env* Testing
Package
P2D
to Deployment
Automated monitoring in Deploy to Production
E
production Env*

30 Copyright ©2014 CollabNet, Inc. All Rights Reserved.

 Phased Approach

2 Multi Layered
COLLABORATION

Teams
Capability Model
Continuous • Our assessment framework is built
Product Teams Release and on this staged approach. Staged
Monitoring approach does necessarily mean
Dev and ops team work that it is a sequence
Multidisciplinary Continuous together seamlessly
Project Teams Testing delivering both projects
• TechM would evaluate organization
and managing
Continuous regression operations. Metrics are as-is status against these stages
reduces the number of
SME Shared Continuous defects; most
captured to monitor
process performance
Teams Delivery integrated bugs are and continuous • As-is assessment would cover
fixed regularly to avoid
Project is able to align
last minute changes;
improvement. people, process, tools, policies, KPIs
Continuous itself with enterprise
test cases reused; Level 5 and flow
Isolated Teams release cadence;
Integration environments are
‘non-functional’ code
quality measured and
provisioned ‘on
technical debt • As an outcome of the assessment
Projects are able demand’; releases
Continuous demonstrate the deployed across
improved we would provide the gaps and the
Level 4
Planning behavior of continuous multiple environments transformation roadmap to cover
Demand pipeline feedback; as opposed including data these gaps
established; Release plans to a periodic schedule, migration
are developed and builds are triggered by Level 3
maintained for individual every commit to the • The Devops consultant/ Program
applications; Projects are single code repository
able to establish
manager would work towards
traceability across Level 2 implementing this transformation
requirements, design, roadmap
development, testing and
defect management is
maintained through out
the lifecycle
Level 1 VALUE

31 Copyright ©2014 CollabNet, Inc. All Rights Reserved.

 Transformation Phases

32 Copyright ©2014 CollabNet, Inc. All Rights Reserved.

 iSPEED tm Methodology

Interview Study Plan Enable and Expand Diffuse

• Interview key • Review historical data,
customer documents, tools and processes
stakeholders • Understand the organization
• Understand customer structure
context • Interview key stakeholders
• Abstract key business using diagnostic tool
drivers • Prepare business case
• Enlist the top • Identify multiple solutions
priorities blueprints
• Prepare transformation
roadmap
• Identify Enterprise Change
Enablement team
• Develop workflows as per
selected blueprint
• Review the workflows with all
stakeholders
• Tune up workflows
• Develop detailed process,
templates, checklists, guidelines
• Modify tool workflows
• Facilitate deployment planning
workshops, prioritize process
activities , prepare deployment
plans
• Enable teams by training and
support
• Identify projects and deploy
• Aggregate data
• Enable Continuous
Improvement
• Quantify benefits and Showcase
projects
• Collate and analyze
performance data
• Benchmark and improve
• Repot
• Establish improvement log
• Continue deployment support

Max 4hrs 2 to 3 WKS 4 to 6 WKS 8 to 12 WKS Ongoing

• Business case • Trainings documents

• Process Workflows • Management reports
Feasibility Decision • Solution blueprint • Project data
• Processes, templates, checklist • Improvement log
• Transformation roadmap • Audit & Management reports

• Fixed Price • Fixed Price

• Fixed Price • Time and Hire • Time and Hire • Time and Hire
• Outcome based pricing • Outcome based pricing

Deliverables
Engagement model

33 Copyright ©2014 CollabNet, Inc. All Rights Reserved.

 DevOps Transformation Assessment (DOT)

ADCM provides a structured and tested

approach for any organizations Devops
strategy. Outcome of the assessment
provides a roadmap that can be used for
Devops strategy and rollout

34 Copyright ©2014 CollabNet, Inc. All Rights Reserved.

 DOT Analytics

Business IT
Alignment • Each stage has multiple focus areas.
4 The depiction here is for continuous
3.5 planning
Task planning 3 Portfolio planning
2.5 • To arrive at the score, TechM
2 consultants would speak to various
1.5 stakeholders and take them through a
1 series of questions
0.5 Value addition
Agile delivery
0 Release planning Collaboration • Each focus area is evaluates for
planning
Institutionalization • the value addition that it is
currently bringing,
• The collaboration it facilitates
• Level of implementation that
has been achieved
Infrastructure
Capability planning
planning
• Focus areas where today nothing exists
is given a score of zero
Capacity planning
• Further for each area a detailed
improvement plan is prepared

Continuous planning

35 Copyright ©2014 CollabNet, Inc. All Rights Reserved.

 DOT Roadmap Sample

Continuous planning W1 W2 W3 W4 W5 W6 W7
People
Process
Business IT Tooling
alignment Flow & integration
Performance & outcomes
Deliverables
People
Process
Portfolio Tooling
Planning Flow & integration
Performance & outcomes
Deliverables
People
Process
Release Tooling
Planning Flow & integration
Performance & outcomes
Deliverables
People
Process
Capability Tooling
Planning Flow & integration
Performance & outcomes
Deliverables
People
Process
Capacity Tooling
Planning Flow & integration
Performance & outcomes
Deliverables
People
Process
Tooling
Infrastructure planning Flow & integration
Performance & outcomes
Deliverables
People
Process
Tooling
Agile delivery planning Flow & integration
Performance & outcomes
Deliverables
People
Process
Task Tooling
planning Flow & integration
Performance & outcomes
Deliverables

36 Copyright ©2014 CollabNet, Inc. All Rights Reserved.

 Thank You
Contact Details:

Kevin Hancock - CollabNet

kevin@collab.net

Sujoy Sen – Tech Mahindra

Sujoy.Sen@TechMahindra.com

37 Copyright ©2014 CollabNet, Inc. All Rights Reserved.

 CollabNet, Inc.
8000 Marina Blvd., Suite 600
Brisbane, CA 94005

www.collab.net © 2014 CollabNet, Inc., All rights reserved. CollabNet is a

trademark or registered trademark of CollabNet Inc., in the US
+1-650-228-2500 and other countries. All other trademarks, brand names, or
+1-888-778-9793 product names belong to their respective holders.
blogs.collab.net
twitter.com/collabnet
www.facebook.com/collabnet
www.linkedin.com/company/collabnet-inc

38 Copyright ©2014 CollabNet, Inc. All Rights Reserved.