Automation and Developer

Infrastructure
Empowering Engineers to Move from Idea to Production
Alex Thurlow
Engineering Manager
I help
peopl
e
get
jobs.
 Unique Visitors (millions) Indeed is the #1
Million unique visitors external source of hire
180 64% of US job searchers search
180 M on indeed each month
160
180
140 million
120 unique users
80.2M
100
unique US visitors per month
80
16M
60 jobs

40 60+
countries
20

0 28
2009 2010 2011 2012 2013 2014 2015 languages
Alex Thurlow
Plus many more!
What should we build?
What’s best for the jobseeker?
How do we know?
Data-Driven Product Design
go.indeed.com/ddpd
 Lean Startup Cycle

Build

Learn Measure
 Deliver

Learn Measure
Data driven product design requires
quick feedback
Engineering Velocity
go.indeed.com/evtt
 Engineering Velocity
go.indeed.com/evtt

Developer Software Release

Productivit Architectur Process
y e
 Engineering Velocity
go.indeed.com/evtt

Developer Software Release

Productivit Architectur Process
y e
Service-Oriented Architecture
Make small, independent applications
 Deliver

Quickl
y!

Learn Measure
 Deliver

Learn Measure
 Engineering Velocity
go.indeed.com/evtt

Developer Software Release

Productivit Architectur Process
y e
Jon Hanks
Software Engineer
I help
peopl
e
get
jobs.
 Hobo
Automated developer provisioning
First goal for a new developer
Push code to production in week 1
 Deliver

Learn Measure
1:1 Mentorship
Setup = 2-3 Days
1-2 days left to get to production
Team Rotation
Helps new developer understand Indeed
Rotate every 4 weeks through 3 teams
Rotate every 4 weeks through 3 teams and face the
same setup challenges in each team.

Frustration!!!
 Experienced devs face a similar
problem working on multiple projects
or switching projects
Service-Oriented Architecture
Search Webapp

Job Database
 Search Webapp

Job Service Resume service Location Service

Job Search Index Resume Database GeoIP Database

 Need to run several
interconnected applications
 Total Developers Producing Code By Month

2014 Apr Jul Oct 2015 Apr Jul Oct 2016

1:1 mentorship didn’t scale
Use technology to speed
developer provisioning
Vagrant automates creating virtualized
development environments
 Virtual Machine

Search
Webapp

Resume Location
Vagrantfi Job Service
service Service
le

Job Search Resume GeoIP

Index Database Database
Success!
(kind of)
 Our Solution—The Good

Reproducible environments
 Our Solution—The Good

Sharable between devs

Our Solution—The Good

<1 day setup time

 Our Solution—The Bad

Slow to create environments. 15 minutes or

1 more!
Hard initial setup especially for large application
2 stacks
3 Virtual machine memory overhead

4 Limited support for different application types

5 Only Linux support

 Our Solution—The Bad

Slow to create environments. 15 minutes or

1 more!
Hard initial setup especially for large application
2 stacks
3 Virtual machine memory overhead

4 Limited support for different application types

5 Only Linux support

Download full machine image
Download and install required software
Start every application
 Our Solution—The Bad

Slow to create environments. 15 minutes or

1 more!
Hard initial setup especially for large application
2 stacks
3 Virtual machine memory overhead

4 Limited support for different application types

5 Only Linux support

 Virtual Machine

Search
Webapp

Resume Location
Vagrantfi Job Service
service Service
le

Job Search Resume GeoIP

Index Database Database
 Virtual Machine Virtual Machine

Search Employer
Webapp Webapp

Job Resume Location Job Resume Location

Service service Service Service service Service

Job Job
Resume GeoIP Resume GeoIP
Search Search
Database Database Database Database
Index Index
 Our Solution—The Bad

Slow to create environments. 15 minutes or

1 more!
Hard initial setup especially for large application
2 stacks
3 Virtual machine memory overhead

4 Limited support for different application types

5 Only Linux support

Shared RAM
Remainin
Virtual
g
Machine
Memory
 Our Solution—The Bad

Slow to create environments. 15 minutes or

1 more!
Hard initial setup especially for large application
2 stacks
3 Virtual machine memory overhead

4 Limited support for different application types

5 Only Linux support

 Our Solution—The Bad

Slow to create environments. 15 minutes or

1 more!
Hard initial setup especially for large application
2 stacks
3 Virtual machine memory overhead

4 Limited support for different application types

5 Only Linux support

We’re all getting Macbooks!
Hooray?
 Our Solution—The Bad

Slow to create environments. 15 minutes or

1 more!
Hard initial setup especially for large application
2 stacks
3 Virtual machine memory overhead

4 Limited support for different application types

5 Only Linux support

Hobo (hō-bō)
synonyms: vagrant
 Hobo goals

1 Support all types of applications

2 Quick startup

3 OS X support

4 Support complex application stacks simply

Simple for developers to use for development and
5 debugging
6 Easy to share
 Hobo goals

1 Support all types of applications

2 Quick startup

3 OS X support

4 Support complex application stacks simply

Simple for developers to use for development and
5 debugging
6 Easy to share
 Hobo goals

1 Support all types of applications

2 Quick startup

3 OS X support

4 Support complex application stacks simply

Simple for developers to use for development and
5 debugging
6 Easy to share
 Hobo goals

1 Support all types of applications

2 Quick startup

3 OS X support

4 Support complex application stacks simply

Simple for developers to use for development and
5 debugging
6 Easy to share
 Hobo goals

1 Support all types of applications

2 Quick startup

3 OS X support

4 Support complex application stacks simply

Simple for developers to use for development and
5 debugging
6 Easy to share
 Hobo goals

1 Support all types of applications

2 Quick startup

3 OS X support

4 Support complex application stacks simply

Simple for developers to use for development and
5 debugging
6 Easy to share
What is Docker?
 Docker Engine

Container 2

Container 3
Container 1
Bins & Bins &
VM VM
Libs Libs

Guest Guest
OS OS

Hypervisor Bins & Libs

Host OS Host OS
Server with Virtual Server with Docker
Machines Containers
Docker terminology

Image
An OS filesystem
 Docker terminology

Container
A running instance of an image
 Docker terminology

Registry
A server for storing images
 Hobo goals

1 Support all types of applications

2 Quick startup

3 OS X support

4 Support complex application stacks simply

Simple for developers to use for development and
5 debugging
6 Easy to share
 Hobo goals

1 Support all types of applications

2 Quick startup

3 OS X support

4 Support complex application stacks simply

Simple for developers to use for development and
5 debugging
6 Easy to share
 Hobo goals

1 Support all types of applications

2 Quick startup

3 OS X support

4 Support complex application stacks simply

Simple for developers to use for development and
5 debugging
6 Easy to share
 Our solution

Create and publish a base image

for each application type
 Java

CentOS
 Java
Publish hobo-
CentOS java
Docker allows “Inheriting” Images
Tomcat

Java

CentOS
 Tomcat
Publish hobo-
hobo-java tomcat
 Ready to use Hobo Images

TOMCAT
 Hobo goals

1 Support all types of applications

2 Quick startup

3 OS X support

4 Support complex application stacks simply

Simple for developers to use for development and
5 debugging
6 Easy to share
 Virtual Machine

Search Webapp

Job Service Resume service Location Service

Job Search Index Resume Database GeoIP Database

Docker is designed to have one
process per container
 Search Webapp

Job Service Resume service Location Service

Job Search Index Resume Database GeoIP Database

 Employer
Search Webapp
Webapp

Job Service Resume service Location Service

Job Search Index Resume Database GeoIP Database

Networking is easy with One machine
Many machines = we need
service discovery
 Service Discovery
Enabling services to find other
services automatically
 What is Consul?

A key/value store geared towards service discovery

Consul has both an HTTP interface and a DNS interface
PUT /v1/catalog/register
{"Node": "user-service", "Address": "10.0.0.3"}

$ dig +short user-service.node.consul

10.0.0.3
Now all config files can reference
other containers by name!
How do we start a stack of multiple
dependent applications?
 Hobo Config File

[HOBO-SETTINGS]
depends = user-service search-service location-service

[job-service]
type = application

[job-service-mysql]
type = resource

[job-service-mongo]
type = resource
 Hobo Config File

[HOBO-SETTINGS]
depends = user-service search-service location-service

[job-service]
type = application

[job-service-mysql]
type = resource

[job-service-mongo]
type = resource
 Hobo Config File

[HOBO-SETTINGS]
depends = user-service search-service location-service

[job-service]
type = application

[job-service-mysql]
type = resource

[job-service-mongo]
type = resource
 Hobo Config File
user-service

[HOBO-SETTINGS]
depends =

[user-service]
type = application

[user-service-mysql]
type = resource
 Hobo goals

1 Support all types of applications

2 Quick startup

3 OS X support

4 Support complex application stacks simply

Simple for developers to use for development and
5 debugging
6 Easy to share
 Hobo goals

1 Support all types of applications

2 Quick startup

3 OS X support

4 Support complex application stacks simply

Simple for developers to use for development and
5 debugging
6 Easy to share
src

test

README

build.xm
l
hobo

src

test

README

build.xm
l
hobo

job-service

job-service-mysql

job-service-mongo

hobo_config
hobo

job-service

config

Dockerfile

run.sh
 Dockerfile

FROM dockerreg.indeed.net/hobo/hobo-tomcat
hobo

job-service

config
tomcat.xm
l
mysql-
conn.properties
job-
service.properties
hobo

job-service

config
tomcat.xm
l
mysql-
conn.properties
job-
service.properties
myconfig

job-
service.properties
hobo

job-service

config

Dockerfile

run.sh
 run.sh
Start/Stop commands
 run.sh
Environment variables
 run.sh
Volume mounts
 What are volume mounts?
They mount parts of the host filesystem
into the container
 What are volume mounts?
They mount parts of the host filesystem
into the container
What do we mount?
What do we mount?
Built code
What do we mount?
Configuration files
What do we mount?
Log directories
What do we mount?
Data directories
 Hobo goals

1 Support all types of applications

2 Quick startup

3 OS X support

4 Support complex application stacks simply

Simple for developers to use for development and
5 debugging
6 Easy to share
 Using Hobo

git clone repository

ant hobo-deploy
Results
 Active Usage Increase Over Time

2014 Apr Jul Oct 2015 Apr Jul Oct 2016 Apr
 330
Projects Configured For Hobo
 “Sometimes I need to run apps from
other teams and the first thing I look
for is ‘Is this hobo’d?’. If so I’m happy!”
- Indeed Developer
15 minutes 30 seconds
 10 times per week means
120 hours per year for each developer
Lessons Learned and Takeaways
 Lessons Learned and Takeaways

Volume mounts are useful

 Lessons Learned and Takeaways

Make it easy to use for the default

case but fully customizable
Lessons Learned and Takeaways

Docker is moving quickly

 Engineering Velocity
go.indeed.com/evtt

Developer Software Release

Productivity Architecture Process
 Control Tower
Automating Release Management
 Record
Develop Features Define Release Test Release
Release
Local Merge desired Build
Check Issue
developmen features to Release in Tag Release
Statuses
t release Jenkins

Check code Run unit Delete

Push Git Upgrade library
review statuses tests in release
Branch dependencies
for issues Jenkins branch

Jenkins
Send bugs Delete
Create Code Mark Issue Check Git branch deploys in
Push to origin back to feature
Review as Resolved merge status QA
developer branches
environment
Yes
Merge
Take Code Check dependent Test release
Update issue Developer release to
Review issues of each in QA
statuses fixes bugs integration
Feedback feature environment
branch

Check Merge
Does the feature Merge
with feature
Creates Pull meet all release Are there release to
develope Create fix version branch to
Request criteria? bugs? production
r release
branch
Yes branch

Add to list of Create deploy Deploy to

Close Issues
features for ticket production
release

Determine which Add fix version to Close Deploy

library upgrades issues in release ticket
are available or
needed

Link other
Release fix
Create release dependent issues
version
branch to deploy ticket
Complex and manual
 Goals of Our Release Process

1 Understand which features are being released

Understand cross-product and cross-team
2 dependencies
3 Quickly address bugs in releases
Record release details for tracking, analysis, and
4 repeatability
 Goals of Our Release Process

1 Understand which features are being released

Understand cross-product and cross-team
2 dependencies
3 Quickly address bugs in releases
Record release details for tracking, analysis, and
4 repeatability
 Goals of Our Release Process

1 Understand which features are being released

Understand cross-product and cross-team
2 dependencies
3 Quickly address bugs in releases
Record release details for tracking, analysis, and
4 repeatability
 Goals of Our Release Process

1 Understand which features are being released

Understand cross-product and cross-team
2 dependencies
3 Quickly address bugs in releases
Record release details for tracking, analysis, and
4 repeatability
 Goals of Our Release Process

1 Understand which features are being released

Understand cross-product and cross-team
2 dependencies
3 Quickly address bugs in releases
Record release details for tracking, analysis, and
4 repeatability
 Goals of Our Release Process

1 Understand which features are being released

Understand cross-product and cross-team
2 dependencies
3 Quickly address bugs in releases
Record release details for tracking, analysis, and
4 repeatability
 Record
Develop Features Define Release Test Release
Release
Local Merge desired Build
Check Issue
developmen features to Release in Tag Release
Statuses
t release Jenkins

Check code Run unit Delete

Push Git Upgrade library
review statuses tests in release
Branch dependencies
for issues Jenkins branch

Jenkins
Send bugs Delete
Create Code Mark Issue Check Git branch deploys in
Push to origin back to feature
Review as Resolved merge status QA
developer branches
environment
Yes
Merge
Take Code Check dependent Test release
Update issue Developer release to
Review issues of each in QA
statuses fixes bugs integration
Feedback feature environment
branch

Check Merge
Does the feature Merge
with feature
Creates Pull meet all release Are there release to
develope Create fix version branch to
Request criteria? bugs? production
r release
branch
Yes branch

Add to list of Create deploy Deploy to

Close Issues
features for ticket production
release

Determine which Add fix version to Close Deploy

library upgrades issues in release ticket
are available or
needed

Link other
Release fix
Create release dependent issues
version
branch to deploy ticket
The life of a feature
Develop Define Test Deploy
Feature Release & Record
We use standard development and
project management tools
&
Git: with great power comes
great complexity
Git flow
master
jira/IND-
123

master
jira/IND-
123

master
 Develop Features
Local
development

Push Git Branch

=
Create Code Mark Issue as
Develop Feature Review Resolved

Take Code Review

Feedback

Creates Pull
Request
Feature code can also be in libraries
 Search Webapp

common-
common-webapp jobservice-api
database
 Develop Features
Local
development

Push Git Branch

=
Create Code Mark Issue as
Develop Feature Review Resolved

Take Code Review

Feedback

Creates Pull
Request
Release Managers Execute Releases
 Goals of Our Release Process

1 Understand which features are being released

Understand cross-product and cross-team
2 dependencies
3 Quickly address bugs in releases
Record release details for tracking, analysis, and
4 repeatability
 Record
Develop Features Define Release Test Release
Release
Local Merge desired Build
Check Issue
developmen features to Release in Tag Release
Statuses
t release Jenkins

Check code Run unit Delete

Push Git Upgrade library
review statuses tests in release
Branch dependencies
for issues Jenkins branch

Jenkins
Send bugs Delete
Create Code Mark Issue Check Git branch deploys in
Push to origin back to feature
Review as Resolved merge status QA
developer branches
environment
Yes
Merge
Take Code Check dependent Test release
Update issue Developer release to
Review issues of each in QA
statuses fixes bugs integration
Feedback feature environment
branch

Check Merge
Does the feature Merge
with feature
Creates Pull meet all release Are there release to
develope Create fix version branch to
Request criteria? bugs? production
r release
branch
Yes branch

Add to list of Create deploy Deploy to

Close Issues
features for ticket production
release

Determine which Add fix version to Close Deploy

library upgrades issues in release ticket
are available or
needed

Link other
Release fix
Create release dependent issues
version
branch to deploy ticket
 Record
Develop Features Define Release Test Release
Release
Local Merge desired Build
Check Issue
developmen features to Release in Tag Release
Statuses
t release Jenkins

Check code Run unit Delete

Push Git Upgrade library
review statuses tests in release
Branch dependencies
for issues Jenkins branch

Jenkins
Send bugs Delete
Create Code Mark Issue Check Git branch deploys in
Push to origin back to feature
Review as Resolved merge status QA
developer branches
environment
Yes
Merge
Take Code Check dependent Test release
Update issue Developer release to
Review issues of each in QA
statuses fixes bugs integration
Feedback feature environment
branch

Check Merge
Does the feature Merge
with feature
Creates Pull meet all release Are there release to
develope Create fix version branch to
Request criteria? bugs? production
r release
branch
Yes branch

Add to list of Create deploy Deploy to

Close Issues
features for ticket production
release

Determine which Add fix version to Close Deploy

library upgrades issues in release ticket
are available or
needed

Link other
Release fix
Create release dependent issues
version
branch to deploy ticket
master

releas
e
 Search Webapp

common-
common-webapp jobservice-api
database
 Search Webapp

common-
common-webapp jobservice-api
database
1.2.6 1.0.1
3.1.5
Build Run Tests Deploy to QA
 Goals of Our Release Process

1 Understand which features are being released

Understand cross-product and cross-team
2 dependencies
3 Quickly address bugs in releases
Record release details for tracking, analysis, and
4 repeatability
What’s in the box?
Look at a dashboard in Jira with everything
that’s supposed to be in the release.
Look at Git log to see commits for features
Look at Git log for each new library version
Look at code reviews for each feature
to make sure they’re complete
 Goals of Our Release Process

1 Understand which features are being released

Understand cross-product and cross-team
2 dependencies
3 Quickly address bugs in releases
Record release details for tracking, analysis, and
4 repeatability
Use Jira for Release Tracking
 Fix Version

Bug Bug Feature

 Deploy
Fix Version
Ticket

Bug Bug Feature

 Deploy
Fix Version
Ticket

Bug Bug Feature

Other App
Feature
 Deploy
Fix Version
Ticket

Bug Bug Feature

Depends
upon

Other App
Feature
 Deploy
Fix Version
Ticket

Bug Bug Feature Depends

upon
Depends
upon

Other App
Feature
Test
 Goals of Our Release Process

1 Understand which features are being released

Understand cross-product and cross-team
2 dependencies
3 Quickly address bugs in releases
Record release details for tracking, analysis, and
4 repeatability
 Record
Develop Features Define Release Test Release Release
Local Merge desired Build
Check Issue
developmen features to Release in Tag Release
Statuses
t release Jenkins

Check code Run unit Delete

Push Git Upgrade library
review statuses tests in release
Branch dependencies
for issues Jenkins branch

Jenkins
Send bugs Delete
Create Code Mark Issue Check Git branch deploys in
Push to origin back to feature
Review as Resolved merge status QA
developer branches
environment
Yes
Merge
Take Code Check dependent Test release
Update issue Developer release to
Review issues of each in QA
statuses fixes bugs integration
Feedback feature environment
branch

Check Merge
Does the feature Merge
with feature
Creates Pull meet all release Are there release to
develope Create fix version branch to
Request criteria? bugs? production
r release
branch
Yes branch

Add to list of Create deploy Deploy to

Close Issues
features for ticket production
release

Determine which Add fix version to Close Deploy

library upgrades issues in release ticket
are available or
needed

Link other
Release fix
Create release dependent issues
version
branch to deploy ticket
Deploy!
 Goals of Our Release Process

1 Understand which features are being released

Understand cross-product and cross-team
2 dependencies
3 Quickly address bugs in releases
Record release details for tracking, analysis, and
4 repeatability
 Record
Develop Features Define Release Test Release Release
Local Merge desired Build
Check Issue
developmen features to Release in Tag Release
Statuses
t release Jenkins

Check code Run unit Delete

Push Git Upgrade library
review statuses tests in release
Branch dependencies
for issues Jenkins branch

Jenkins
Send bugs Delete
Create Code Mark Issue Check Git branch deploys in
Push to origin back to feature
Review as Resolved merge status QA
developer branches
environment
Yes
Merge
Take Code Check dependent Test release
Update issue Developer release to
Review issues of each in QA
statuses fixes bugs integration
Feedback feature environment
branch

Check Merge
Does the feature Merge
with feature
Creates Pull meet all release Are there release to
develope Create fix version branch to
Request criteria? bugs? production
r release
branch
Yes branch

Add to list of Create deploy Deploy to

Close Issues
features for ticket production
release

Determine which Add fix version to Close Deploy

library upgrades issues in release ticket
are available or
needed

Link other
Release fix
Create release dependent issues
version
branch to deploy ticket
jira/IND-
123

master

release
jira/IND-
123

master

deploy/2016-06-
22
jira/IND-
123

master

deploy/2016-06-
22
 master

deploy/2016-06-
22
 JIRA updates
Close released issues
 JIRA updates
Close deploy ticket
 JIRA updates
Mark Fix Version as “Released”
 Record
Develop Features Define Release Test Release Release
Local Merge desired Build
Check Issue
developmen features to Release in Tag Release
Statuses
t release Jenkins

Check code Run unit Delete

Push Git Upgrade library
review statuses tests in release
Branch dependencies
for issues Jenkins branch

Jenkins
Send bugs Delete
Create Code Mark Issue Check Git branch deploys in
Push to origin back to feature
Review as Resolved merge status QA
developer branches
environment
Yes
Merge
Take Code Check dependent Test release
Update issue Developer release to
Review issues of each in QA
statuses fixes bugs integration
Feedback feature environment
branch

Check Merge
Does the feature Merge
with feature
Creates Pull meet all release Are there release to
develope Create fix version branch to
Request criteria? bugs? production
r release
branch
Yes branch

Add to list of Create deploy Deploy to

Close Issues
features for ticket production
release

Determine which Add fix version to Close Deploy

library upgrades issues in release ticket
are available or
needed

Link other
Release fix
Create release dependent issues
version
branch to deploy ticket
 Goals of Our Release Process

1 Understand which features are being released

Understand cross-product and cross-team
2 dependencies
3 Quickly address bugs in releases
Record release details for tracking, analysis, and
4 repeatability
 Problems

1 Release management took a lot of time

It was hard to understand what exactly was in a
2 release
3 Lots of manual steps = possibility for error
Only senior engineers knew enough to handle
4 releases
 Problems

1 Release management took a lot of time

It was hard to understand what exactly was in a
2 release
3 Lots of manual steps = possibility for error
Only senior engineers knew enough to handle
4 releases
 Problems

1 Release management took a lot of time

It was hard to understand what exactly was in a
2 release
3 Lots of manual steps = possibility for error
Only senior engineers knew enough to handle
4 releases
 Problems

1 Release management took a lot of time

It was hard to understand what exactly was in a
2 release
3 Lots of manual steps = possibility for error
Only senior engineers knew enough to handle
4 releases
Release with 4 features

>100
Clicks and Git Commands
 ~13
Extra Actions Per Feature
 Total Developers Producing Code By Month

2014 Apr Jul Oct 2015 Apr Jul Oct 2016

 Unique projects deployed per month

2014 Apr Jul Oct 2015 Apr Jul Oct 2016

We needed to automate
 Why automate rather than simplify?

Data History Transparency

We needed to automate
Control Tower
 Goals of Our Release Process

1 Understand which features are being released

Understand cross-product and cross-team
2 dependencies
3 Quickly address bugs in releases
Record release details for tracking, analysis, and
4 repeatability
 Goals of Our Release Process

1 Understand which features are being released

Understand cross-product and cross-team
2 dependencies
3 Quickly address bugs in releases
Record release details for tracking, analysis, and
4 repeatability
What feature code is in libraries?
 Indeed Webapp

common-
common-webapp jobservice-api
database
1.2.6 1.0.1
3.1.5
 Goals of Our Release Process

1 Understand which features are being released

Understand cross-product and cross-team
2 dependencies
3 Quickly address bugs in releases
Record release details for tracking, analysis, and
4 repeatability
 Record
Develop Features Define Release Test Release
Release
Local Merge desired Build
Check Issue
developmen features to Release in Tag Release
Statuses
t release Jenkins

Check code Run unit Delete

Push Git Upgrade library
review statuses tests in release
Branch dependencies
for issues Jenkins branch

Jenkins
Send bugs Delete
Create Code Mark Issue Check Git branch deploys in
Push to origin back to feature
Review as Resolved merge status QA
developer branches
environment
Yes
Merge
Take Code Check dependent Test release
Update issue Developer release to
Review issues of each in QA
statuses fixes bugs integration
Feedback feature environment
branch

Check Merge
Does the feature Merge
with feature
Creates Pull meet all release Are there release to
develope Create fix version branch to
Request criteria? bugs? production
r release
branch
Yes branch

Add to list of Create deploy Deploy to

Close Issues
features for ticket production
release

Determine which Add fix version to Close Deploy

library upgrades issues in release ticket
are available or
needed

Link other
Release fix
Create release dependent issues
version
branch to deploy ticket
 Record
Develop Features Define Release Test Release
Release
Local Merge desired Build
Check Issue
developmen features to Release in Tag Release
Statuses
t release Jenkins

Check code Run unit Delete

Push Git Upgrade library
review statuses tests in release
Branch dependencies
for issues Jenkins branch

Jenkins
Send bugs Delete
Create Code Mark Issue Check Git branch deploys in
Push to origin back to feature
Review as Resolved merge status QA
developer branches
environment
Yes
Merge
Take Code Check dependent Test release
Update issue Developer release to
Review issues of each in QA
statuses fixes bugs integration
Feedback feature environment
branch

Check Merge
Does the feature Merge
with feature
Creates Pull meet all release Are there release to
develope Create fix version branch to
Request criteria? bugs? production
r release
branch
Yes branch

Add to list of Create deploy Deploy to

Close Issues
features for ticket production
release

Determine which Add fix version to Close Deploy

library upgrades issues in release ticket
are available or
needed

Link other
Release fix
Create release dependent issues
version
branch to deploy ticket
 Goals of Our Release Process

1 Understand which features are being released

Understand cross-product and cross-team
2 dependencies
3 Quickly address bugs in releases
Record release details for tracking, analysis, and
4 repeatability
 Record
Develop Features Define Release Test Release Release
Local Merge desired Build
Check Issue
developmen features to Release in Tag Release
Statuses
t release Jenkins

Check code Run unit Delete

Push Git Upgrade library
review statuses tests in release
Branch dependencies
for issues Jenkins branch

Jenkins
Send bugs Delete
Create Code Mark Issue Check Git branch deploys in
Push to origin back to feature
Review as Resolved merge status QA
developer branches
environment
Yes
Merge
Take Code Check dependent Test release
Update issue Developer release to
Review issues of each in QA
statuses fixes bugs integration
Feedback feature environment
branch

Check Merge
Does the feature Merge
with feature
Creates Pull meet all release Are there release to
develope Create fix version branch to
Request criteria? bugs? production
r release
branch
Yes branch

Add to list of Create deploy Deploy to

Close Issues
features for ticket production
release

Determine which Add fix version to Close Deploy

library upgrades issues in release ticket
are available or
needed

Link other
Release fix
Create release dependent issues
version
branch to deploy ticket
 jira/IND-123

master

release
IND-123 commits In release IND-123 commit Not in release

jira/IND-123

master

release
 jira/IND-123

master

release
 Goals of Our Release Process

1 Understand which features are being released

Understand cross-product and cross-team
2 dependencies
3 Quickly address bugs in releases
Record release details for tracking, analysis, and
4 repeatability
 Record
Develop Features Define Release Test Release Release
Local Merge desired Build
Check Issue
developmen features to Release in Tag Release
Statuses
t release Jenkins

Check code Run unit Delete

Push Git Upgrade library
review statuses tests in release
Branch dependencies
for issues Jenkins branch

Jenkins
Send bugs Delete
Create Code Mark Issue Check Git branch deploys in
Push to origin back to feature
Review as Resolved merge status QA
developer branches
environment
Yes
Merge
Take Code Check dependent Test release
Update issue Developer release to
Review issues of each in QA
statuses fixes bugs integration
Feedback feature environment
branch

Check Merge
Does the feature Merge
with feature
Creates Pull meet all release Are there release to
develope Create fix version branch to
Request criteria? bugs? production
r release
branch
Yes branch

Add to list of Create deploy Deploy to

Close Issues
features for ticket production
release

Determine which Add fix version to Close Deploy

library upgrades issues in release ticket
are available or
needed

Link other
Release fix
Create release dependent issues
version
branch to deploy ticket
 Record
Develop Features Define Release Test Release Release
Local Merge desired Build
Check Issue
developmen features to Release in Tag Release
Statuses
t release Jenkins

Check code Run unit Delete

Push Git Upgrade library
review statuses tests in release
Branch dependencies
for issues Jenkins branch

Jenkins
Send bugs Delete
Create Code Mark Issue Check Git branch deploys in
Push to origin back to feature
Review as Resolved merge status QA
developer branches
environment
Yes
Merge
Take Code Check dependent Test release
Update issue Developer release to
Review issues of each in QA
statuses fixes bugs integration
Feedback feature environment
branch

Check Merge
Does the feature Merge
with feature
Creates Pull meet all release Are there release to
develope Create fix version branch to
Request criteria? bugs? production
r release
branch
Yes branch

Add to list of Create deploy Deploy to

Close Issues
features for ticket production
release

Determine which Add fix version to Close Deploy

library upgrades issues in release ticket
are available or
needed

Link other
Release fix
Create release dependent issues
version
branch to deploy ticket
 Record
Develop Features Define Release Test Release Release
Local Merge desired Build
Check Issue
developmen features to Release in Tag Release
Statuses
t release Jenkins

Check code Run unit Delete

Push Git Upgrade library
review statuses tests in release
Branch dependencies
for issues Jenkins branch

Jenkins
Send bugs Delete
Create Code Mark Issue Check Git branch deploys in
Push to origin back to feature
Review as Resolved merge status QA
developer branches
environment
Yes
Merge
Take Code Check dependent Test release
Update issue Developer release to
Review issues of each in QA
statuses fixes bugs integration
Feedback feature environment
branch

Check Merge
Does the feature Merge
with feature
Creates Pull meet all release Are there release to
develope Create fix version branch to
Request criteria? bugs? production
r release
branch
Yes branch

Add to list of Create deploy Deploy to

Close Issues
features for ticket production
release

Determine which Add fix version to Close Deploy

library upgrades issues in release ticket
are available or
needed

Link other
Release fix
Create release dependent issues
version
branch to deploy ticket
Develop Define Test Deploy
Feature Release
Results
 Problems

1 Release management took a lot of time

It was hard to understand what exactly was in a
2 release
3 Lots of manual steps = possibility for error
Only senior engineers knew enough to handle
4 releases
Release with 4 features

>100
Clicks and Git Commands
Release with 4 features

<15
Control Tower Clicks
 ~13
Extra Actions Per Feature
 0
Extra Actions Per Feature
 Deploys per month

Jul Oct 2015 Apr Jul Oct 2016 Apr

1.9X
Deploys Per Project
 Problems

1 Release management took a lot of time

It was hard to understand what exactly was in a
2 release
3 Lots of manual steps = possibility for error
Only senior engineers knew enough to handle
4 releases
 Problems

1 Release management took a lot of time

It was hard to understand what exactly was in a
2 release
3 Lots of manual steps = possibility for error
Only senior engineers knew enough to handle
4 releases
More knowledge = Less risk
 57 %
of developers are release managers
~3 months
To start doing releases
Lessons Learned and Takeaways
 Lessons Learned and Takeaways

Everything can be automated

 Lessons Learned and Takeaways

Engineers don’t like mystery

 Lessons Learned and Takeaways

Automate the process you have, not

the process you want
Where do we go from here?
Too much human interaction
Train machines to handle releases
 Learn More

Engineering blog & talks -

indeed.tech

Open Source -
opensource.indeedeng.io

Careers - indeed.jobs

Twitter - @IndeedEng