Renew The IT Curriculum

Renewing the IT Curriculum:
Responding to Agile, DevOps, and Digital Transformation

November 1, 2016
This report is an initiative of Advance IT Minnesota,

the Minnesota State Center of Excellence in IT Education
This report is dedicated to the work and vision of the
Minnesota Educational Computing Consortium, 1973-1991
Renewing the IT Curriculum Page |i

Praise for this report
DevOps and Agile are transforming the technology workforce. Of the 8 million developers and 8
million operations professionals employed globally, far too many are at risk of being left behind.
Initiatives like the Dynamic IT curricula project are critical to ensure that technology professionals
have the skills needed in the marketplace, and I'm thrilled to be a part of this effort.
Gene Kim, co-author of "DevOps Handbook" and "The Phoenix Project"
The pace of innovation continues to push IT practitioners, developers, and product management leaders to
embrace new processes and practices like Agile and DevOps as a better way of doing business. Higher
education has a responsibility not only to catch up but lead the way! This work is a critical first step in
charting the future direction and possibilities for how to embed these important concepts in our programs.
Wilson Garland, Executive Director, Advance IT Minnesota
Everything is going digital these days, making application development & programming skills
vital to nearly every sector of the economy. Education must keep up by bringing these skills into
mainstream curriculum for all students and ensuring specialized IT programs include modern
practices in Agile product management, cloud and distributed computing, and DevOps.
Mary Mosman, Hennepin Technical College
Today's businesses are run by technology, and the transformation we are seeing is drastically different from
the landscape five or ten years ago. As a former professor, I understand the hesitation to jump on the latest
bandwagon, but the principles of Agile and DevOps are more than fads -- they are methodologies that
transcend the latest technology or tooling. We must adapt our curriculum to equip our students so they can
be a part of this exciting new world and contribute, innovate, and inspire in this new technical economy.
Nicole Forsgren, Ph.D., DevOps Research and Assessment
Agile, DevOps and the Cloud are transforming IT writ large. I see it every day in my personal
and professional lives. This innovative curriculum report contains the energy and exposition
required to move the academy's needle and help make these incredibly important subjects
mandatory curricula for the 21st century IT work force.
Jeff Sussna, Author, Designing Delivery
“Over the last couple of decades, there has been much progress in our understanding of the
principles and practices of software and IT management. This report is great step forward in
bringing computing and IT curriculum into the 21st century.”
Murray Cantor, Ph.D.
Telling people that they need to see things differently and guiding them on their journey are two
different things. This report provides that guide to the folks that are on the front line of preparing
the next generation workforce while building on their current base of knowledge.
Mary Texer, Blue Cross-Blue Shield MN
Renewing the IT Curriculum Page |i

As a computer science instructor in a publically funded university, it is imperative that we prepare
our students to deliver sustainable, quality-focused software as quickly as possible. This is not just
hype. It is essential to their success. It is essential to our business partners, and finally, it is essential
to our taxpayers and to our economy.
David Levitt, Metropolitan State University
Digital transformation has swept through the economy, disrupting or revolutionizing one industry
after another. Successful transformation, however, is far from guaranteed. Organizations that
attempt digital transformation without the right people and practices risk falling behind. The
curricula promoted by Dynamic IT Education promise to produce a workforce with the DevOps
skills and the foundational Agile knowledge to help organizations across Minnesota survive and
thrive through their digital transformations.
Tom O’Neill, CEO, The Nerdery
Every company today is a software company, and the most successful companies operate as learning
organizations with a customer centric mindset. To provide students with the skills and values needed
to be successful in the marketplace, the educational system needs to keep pace by mirroring industry in
what is offered to students. With software intertwined into every aspect of our society, students need to
understand not only technical aspects, but also design thinking and product management concepts and
practices. Without this updated curriculum to reflect how the world is working today, students will
graduate with fewer opportunities for employment and a lower probability of career success.
Aric Aune, Target Corporation
Every company needs IT to get things done.

Higher-education needs to prepare students for this exciting reality.
The ‘Dynamic IT education’ report provides an excellent platform on which colleges and universities
can build a rigorous and relevant curriculum that prepares students for a rewarding, life-long career
in IT management.
David Yates, Ph.D., Bentley University & Founder, UniOps.org
Today's students may never encounter a physical server, network switch, or storage device during
their future professional careers, and yet they will build more IT infrastructure than all of their
predecessors combined. We need to teach students how to create infrastructure using code and
how to work together collaboratively to manage complex distributed applications.
Jason Baker, University of St.Thomas
Falling cost and complexity barriers to software creation and deployment are enabling
businesses to shift to a 'digital first" strategy, and industry looks to academia to develop curricula
that integrate current industry practices around Agile, DevOps and IT management in order to
educate the next generation IT workforce.
Dave Lounsbury, CTO, The Open Group
Renewing the IT Curriculum P a g e | ii

Agile, DevOps and the Cloud are transforming IT writ large. I see it every day in my personal
and professional lives. This innovative curriculum report contains the energy and exposition
required to move the academy's needle and help make these incredibly important subjects
mandatory curricula for the 21st century IT work force.
Tom McGonagle, F5 Systems & Founder, UniOps.org
Self-organizing cross-functional teams often collaborate to develop the requirements of, and solutions
to software products. All computing and information technology students ought to learn the concepts of
adaptive planning, evolutionary growth, unceasing enhancement, timely delivery, and fast and supple
response to software modification that agile software development promotes. To be prepared for the
future IT workforce, students have to absorb the hands-on skills for secure software development and
operations –we need to teach the students how to run software products from inception to retirement.
Amos Olagunju, St Cloud State University
The digital curriculum is an invaluable resource for implementing DevOps principles in the
classroom. This toolkit lets an instructor bring the disciplines of Quality Assurance, Information
Technology, and Software Development into a single course, providing students with a classroom
experience that is much closer to the real-world business environment.
Mary Lebens, Anoka Technical College
Many times, as educators, we feel that we should do a better job in catching up with the trends in
the industry to prepare our students for their careers. This Dynamic IT curriculum document is a
collaborative effort by educators to connect with the latest trends and to provide students with
tools to navigate through the fast changing digital world. This document has the potential to
make significant changes in how we prepare our future generations to serve the world.
Halbana Tarmizi, Bemidji State University
Organizations rely on technology's power, speed, and agility to sustain a competitive advantage, and
IT professionals must have the requisite knowledge and skills to meet these demands now and into
the future. This report provides higher education the foundations necessary to transform curriculum
and deliver students capable of understanding and meeting these dynamic organizational needs.
Justin Opatrny, Minneapolis Community & Technical College
There is now rapid adoption of dynamically configurable, hosted infrastructure coupled with
truly responsive software development supporting business processes across platforms.
This is a paradigm shift that challenges current educational approaches, necessitating change.
Firasat Khan, Advance IT Minnesota and Metropolitan State University
Digital transformation is upon us. To enable it, DevOps and Agile have matured from once-obscure
alternative approaches into the mainstream consensus. We now face the challenge of creating the
workforce. This report is intended to support educators and students of all kinds in their digital journey.
Charles Betz, Digital Management Academy
Renewing the IT Curriculum P a g e | iii

Foreword
The process of building and maintaining great software is undergoing a number of
seismic shifts, opening up incredible opportunities for entrepreneurs and enabling a
stunning array of internet services. The convergence of Cloud Infrastructure, Software as
a Service, DevOps culture, and Agile methodologies has fundamentally altered software
development for the better. This convergence has also enabled the digital transformation
of entire industries and the process is only accelerating. Finding software engineers and
other IT professionals with the technical skills for digital transformation is a challenge in
and of itself. Finding people who also understand the organizational challenges of
transformation is harder yet. Organizations that rise to the challenge will be the ones that
thrive and grow in the digital economy.
The curricula recommended in this document represent a fantastic realization of the
principlesand technologies that are relevant to the software industry in 2016. A student
that completes the coursework described here will be well prepared to hit the ground
running at a company like The Nerdery, which is high praise. Recent college graduates
typically take a year or more to learn a fraction of the real-world knowledge that this
program will provide.
Of course, it’s impossible to find agents of digital transformation where they don’t exist.
The Nerdery has always prided itself on its ability to find the best Nerds in Minnesota,
and the Minnesota State colleges and universities have always been key sources of talent.
By adopting the recommendations in this document, the Minnesota State System will
produce the agents of digital transformation that companies like The Nerdery needs. We
can’t wait to see what they do.
Renewing the IT Curriculum P a g e | iv

Task force
Coordinators
Charles Betz, University of St. Thomas
Firasat Khan, Metropolitan State University
Authors/Working group
David Bahn, Metropolitan State University
Charles Betz, University of St. Thomas
Svetlana Gluhova, Hennepin Technical College
Firasat Khan, Metropolitan State University
Mary Lebens, Anoka Technical College
Patrick Paulson, Winona State University
Amos Olagunju, St. Cloud State University
Justin Opatrny, Minneapolis Community & Technical College
Grant Spencer, Metropolitan State University
Halbana Tarmizi, Bemidji State University
Advisory group
Bob Anderson, Amazon
Aric Aune, Target Corporation & University of St. Thomas
Jason Baker, University of St. Thomas
Murray Cantor, Ph.D., Aptage
Andrew Chapman, Waiariki Polytechnic, New Zealand
Nicole Forsgren, Ph.D., DevOps Research and Assessment
Lorin Hochstein, Netflix
Gene Kim, Author, The Phoenix Project; The DevOps Handbook
Dave Lounsbury, CTO, The Open Group
David Levitt, Metropolitan State University
Tom McGonagle, F5 & UniOps.org
Heather Mickman, Target Corporation
Jeff Sussna, Author, Designing Delivery
Jim Sutton, NetObjectives
Mary Texer, Blue Cross/Blue Shield MN
David Yates, Ph.D., Bentley University & UniOps.org
Acknowledgements
Thanks to Eri Fujieda of Winona State University Institutional Research for work on the survey.
Document control and citation

This document is the compilation of a series of collectively-authored Github Wiki pages from the
https://github.com/MN-ASG/digi-curr site.
This work belongs to the Authors and Advance IT Minnesota the Minnesota State Center of
Excellence in IT Education. Shared under CreativeCommons Attribution-NonCommercial-
NoDerivs license allowing unmodified and noncommercial copying and distribution.
The following is a recommended citation for this report:
Renewing the IT Curriculum: Responding to Agile, DevOps, and Digital Transformation (Rep.).
(November 1, 2016). St. Paul, MN: Advance IT Minnesota. Retrieved from www.DynamicIT.education
Renewing the IT Curriculum Page |v

Contents
Praise for this report ......................................................................................................................... i
Foreword ........................................................................................................................................ iv
Task force.........................................................................................................................................v
Contents ......................................................................................................................................... vi
Executive Summary .......................................................................................................................1
Introduction ....................................................................................................................................2
Background of this report .......................................................................................................5
Goals of this report .................................................................................................................5
Approach ................................................................................................................................6
Survey ....................................................................................................................................6
Document Structure ................................................................................................................7
Agile, Digital, DevOps: What do we mean? ................................................................................8
Agile and related digital trends ...............................................................................................8
Defining “IT,” “Agile,” “DevOps,” and “digital” ................................................................11
Competency areas .................................................................................................................14
Approach to learning objectives ...........................................................................................15
CA 1 – Dynamic infrastructure and operations ........................................................................16
Virtualization ........................................................................................................................17
Cloud ..................................................................................................................................18
Software Defined Infrastructure ...........................................................................................19
Site reliability engineering....................................................................................................21
CA 2 – Continuous delivery ........................................................................................................23
Agile and related software development practices ...............................................................23
Continuous Delivery pipelines .............................................................................................24
CA 3 – Product management ......................................................................................................27
Digital product management origins and influences ............................................................27
Organizing and managing the product team .........................................................................28
Managing the digital product lifecycle .................................................................................29
Product portfolio management .............................................................................................30
CA 4 – Resource and execution management ...........................................................................32
Principles for executing and scaling .....................................................................................32
Industry practices for execution ...........................................................................................34
CA 5 – Organization and culture ...............................................................................................36
Organizing for digital delivery .............................................................................................36
Enabling people and understanding culture..........................................................................37
Curricula, courses, and pedagogy ..............................................................................................40
Section objectives .................................................................................................................40
Current challenges ................................................................................................................40
Course-specific recommendations ........................................................................................40
Digital labs and simulations .................................................................................................42
Appendix A. Open issues .............................................................................................50
Appendix B. References ..............................................................................................51
Renewing the IT Curriculum P a g e | vi

Executive Summary
Digital transformation requires new approaches to curricula and instruction. New
delivery approaches such as Agile development, Lean Product Management, Site
Reliability Engineering, DevOps, and related trends require an updated educational
response. A survey of educational and business professionals on workforce needs related
to Agile, DevOps, and other topics inform this report, which identifies five competency
areas IT, IS, and computing educators should consider in designing next-generation
curricula:
 Dynamic infrastructure and operations
 Continuous delivery
 Product management
 Resource and execution management
 Organization and culture
These areas are elaborated with definitions, categories, competencies, and sample
learning objectives, with references. Recommendations for adapting existing courses and
practical guidance on developing digital labs and simulations are also included.
Renewing the IT Curriculum Page |1

Introduction
Information technology increasingly pervades economy and society. In the words of
venture capitalist and World Wide Web pioneer Mark Andreessen, "Software is eating
the world."1 Significant changes are apparent in how computing and technology work is
performed. As General Electric CIO Jim Fowler notes:
“When I am in business meetings, I hear people talk about digital as a
function or a role. It is not. Digital is a capability that needs to exist in
every job. Twenty years ago, we broke ecommerce out into its own
organization, and today ecommerce is just a part of the way we work.
That’s where digital and IT are headed; IT will be no longer be a distinct
function, it will just be the way we work.
[W]e’ve moved to a flatter organizational model with "teams of teams"
who are focused on outcomes. These are colocated groups of people who
own a small, minimal viable product deliverable that they can produce in
90 days. The team focuses on one piece of work that they will own
through its complete lifecycle…in [the “back office”] model, the CIO
controls infrastructure, the network, storage, and makes the PCs run.
The CIOs who choose to play that role will not be relevant for long…”2
Fowler echoes the strategic concerns and directions of leading CIOs around the world.
And beneath his words lie fundamentally new approaches to delivering IT and digital
value. These approaches carry various names, but the term “Agile” is prominent, with a
number of variants and extensions (DevOps, Lean IT, Lean Product Development, Web-
scale IT, Lean UX, and so forth). These trends, decades in the making, cannot be
dismissed as "flavor of the month" or a fad, particularly when they are fundamentally
changing the way software is made, and therefore, changing the industry.
The new IT-based digital economy is informed top to bottom by an ecosystem of ideas
founded on Agile software development. This is accompanied by rethinking or outright
rejection of certain practices, including waterfall software development, lengthy “analysis
and design” phases, stage-gated project management, and the use of process
improvement frameworks. Many are introduced to such practices via formal education,
where they are still often presented uncritically.
But as noted by Rigby, Sutherland, and Takeuchi in The Harvard Business Review,
“Now agile methodologies—which involve new values, principles,
practices, and benefits and are a radical alternative to command-and-
control-style management—are spreading across a broad range of
industries and functions and even into the C-suite”. 3
1
(Andreessen 2011)
2
(Heller 2016)
3
(Rigby et al. 2016)

These methods represent an ecosystem extending well beyond its origins in software
development, into questions of product management, operations, and even organization
and culture. Their proven effectiveness in managing many forms of complexity,
including problems not usually seen as “IT” or even “digital,”4 leads to broader and
broader appetite for appropriate instruction and curricula.
Technologies come and go, and it is appropriate that education professionals focus on
underlying principles and practices. Curricula are expensive to change, and commercial
training exists for good reason as stated in the ACM/IEEE/AIS publication Computing
Curricula 2005: The Overview Report:
"Institutions tend to be cautious and conservative, and the complex
nature of academic degree programs means that it is difficult to
implement significant changes rapidly… the pace of change in
computing is quite rapid, while the pace of institutional change generally
can be quite slow . . ."5
However, we are not faced today with merely another ephemeral technology cycle.
Rather, we are experiencing a generational change in foundational models of IT-related
delivery and execution, in response to accelerating digital transformation.
Jeff Sussna, a Minnesota digital professional and author of Designing Delivery, notes that
Digital infusion has fully blossomed. The word infusion refers to the fact
that computer systems are no longer separate from anything else we do.
The digital realm is infusing the physical realm, like tea in hot water…In
order to serve this newly infused world, companies need to undertake
equally deep internal transformations. IT used to be a purely internal
corporate function. IT might impact internal operations efficiency, but
from the consumer’s perspective, it remained invisible…In a digitally
infused business, … IT becomes an integral part
of the customer-facing service…6
Startups; large, established firms; and now mid-size companies are embracing the
insights and practices of digital transformation, and local workforce requirements are
shifting. We believe that academic systems and curriculum owners need up to date
understanding and relevant, mainstream offerings appropriate to these trends.
In sum, we are at an exciting time in the industrialization of digital technology and its
management. All industries, including those that have not been viewed as purely
“technical” are now being challenged by the tsunami known as “digital transformation.”
Diomidis Spinellis, editor in chief of IEEE Software, observed last year that
4
(Spinellis 2015)
5
(Shackelford et al. 2005)
6
(Sussna 2015), pp. 7-8.

…other industries are also producing what’s in effect software
(executable knowledge) but not treating it as such…Although many
industries have developed their own highly effective processes over the
years, software engineering maintains an essential advantage. It has
developed methods and tools that let even small teams manage extremely
high complexity. … This advantage is important because the complexity
in non-software activities is also increasing inexorably… [T]he time has
come to transform our world… by giving back to science and technology
the knowledge software engineering has produced.7
As Spinellis notes, digital tools and techniques are influencing not just IT, but business
practices worldwide. We need new approaches to instructing the next generation
workforce in these topics.
Educational transitions are never easy nor cheap. But meeting the demands of the digital
economy with an appropriately skilled workforce has always been a mission of higher
education. We thank readers for their attention and look forward to continued dialog.
7
(Spinellis 2015)

Background of this report
The Minnesota State system8 is the 5th-largest publicly-funded higher education entity in
the United States, serving 400,000 students, with an annual budget of $1.9 billion. It
represents 24 two-year colleges and 7 universities, based on 54 campuses across the state.
Advance IT Minnesota (http://advanceitmn.org/), the Minnesota State Center of
Excellence in IT and security education, convenes the annual New Directions in IT
Education conference and invites faculty members across the state to identify emerging
curricular needs and propose collaborative projects to develop solutions to specific issues.
At the 2016 New Directions in IT Education Conference, a faculty working group
convened to consider the impacts of current technological trends on IT education. In
particular, working group members were aware of and interested in the following topics:
 Agile software development
 Lean IT
 Cloud and web-scale IT
 DevOps
 Digital transformation
There is increasing evidence that these topics are significant to employers, and call for
some educational response. This document examines these trends in light of current
curricula, and makes recommendations in terms of learning objectives, current courses,
and programmatic initiatives.
Goals of this report

This report is intended as a reference for developing next generation computing-related
curricula appropriate to the new digital economy. It is not itself a curriculum. The goal of
the report is to first and foremost benefit students of all backgrounds and ages, by
translating valuable current practices into a framework, with example learning objectives.
By doing so, their educational outcomes will be both rigorous and career-relevant, and
employers will find them ready to participate in the digital economy.
For faculty and institutions, the intent of this document is to provide a space for
experimentation and inquiry. The Agile movement underlying today’s digital
transformation emphasizes experimentation and self-organization. It would be ironic and
presumptuous to attempt to mandate a standard "Agile Curriculum."
These new developments are broad and complex. Our hope is that this report will be of
service to faculty who are starting to hear from local advisory boards and students about
these new methods and approaches.
8
Formerly the Minnesota State Colleges and Universities system, MSCU

Approach
The primary developers of collegiate curricula guidance are the Association for
Computing Machinery (ACM), the Computer Society of the IEEE, and the Association
for Information Systems. A master list of this guidance can be found at
http://www.acm.org/education/curricula-recommendations
This discussion is not intended to be a comprehensive disciplinary model curriculum
(e.g., Information Systems or Information Technology as a whole), but rather highlights
certain topics of particular concern.
For a framework, we have adopted the same approach as the draft MSIS 2016 model
curriculum.9 While attention to specific approaches is necessary, we also have
incorporated deeper learning objectives regarding principles, and theoretical references
where appropriate. This is not intended as purely applied, vocational curricula guidance.
Figure 1. Framework approach
As with that guidance, the focus is on the program area (competencies and their
categories and areas). Currently we are incorporating example granular learning
objectives under the competencies.
We are not proposing a master, normative set of learning objectives. We share the MSIS
2016 approach that learning objectives are properly the domain of course development.
The example learning objectives are intended to validate the framework, and provide a
useful starting point for course development.
Survey
An Agile/Digital survey was developed with the assistance of Winona State University.
159 survey responses included:
 96 Industry professionals
 35 Academic professionals
 28 working in both industry and academia
60% were from the state of Minnesota.
Some findings relevant to this report’s recommendations are:
9
(Topi et al. 2016), p.2.

 Agile skills are increasingly important in hiring decisions (62% of industry
professionals consider as a factor, and 41% started doing so within the last 3
years)
 59% of industry respondents say that their software development and digital
pipeline are based on Agile/DevOps/Cloud practices
 There is concern about the readiness of the Minnesota workforce for digital
transformation, and the capacity of the state’s educational systems to produce this
workforce (only 32% agree that the workforce is well prepared in terms of
competencies, and 49% disagree that the educational system is producing a large
enough digital workforce).
The complete survey is included in the appendix.
Document Structure
The remainder of this document is organized as follows
Section Description
Agile, Digital, DevOps: what do Provides a reference of how common agile terminology is used in the
we mean? context of this paper.
Digital Competency Areas Provides an overview of the competency areas for digital curricula.
CA1 –Dynamic Infrastructure and Virtualization, cloud, infrastructure as code, site reliability engineering.
Operations
CA2 – Continuous Delivery DevOps pipelines, automation, continuous integration.
CA3 – Product Management Outcomes, value, discovery, exploration, intent, collaboration, Scrum.
CA4 – Resource and Execution Lean product management concerns, queuing, cadence, Kanban.
Management
CA5 – Organization and Culture Current approaches to solving matrix problem, importance of
collaborative, blameless culture.
Curricula Recommendations Adapting current courses; digital labs and simulations.
Open issues Included open issues section for further iterations.

Agile, Digital, DevOps: What do we mean?
Agile and related digital trends
Much of the impetus behind the current trends can be traced to the evolution of software
development philosophy and practice. Over the past fifty years, there have been two
schools of software development management:
 Process control
 Incremental reduction of uncertainty.
The process control method was promoted by the influential CMM author Watts
Humphrey. His vision was to apply statistical process control to the software process. As
he stated at the time:
Dr. W. E. Deming, in his work with the Japanese after World War II,
applied the concepts of statistical process control to many of their
industries. While there are important differences, these concepts are just
as applicable to software as they are to producing consumer goods like
cameras, television sets, or automobiles.10
This point of view was reinforced by stage-gated project management,11 “waterfall”
methodologies, and the methods of leading consultancies.
The second method, often termed “iterative and incremental,” was in evidence as early as
the 1960s12, promoted by Tom Gilb,13 Grady Booch,14 and others. It has had various
incarnations and expressions such as Boehm’s Spiral model,15 the Unified Process and
Xtreme Programming. 16 This approach has continued to gain momentum as the industry
has observed the failures of the first approach.
10
(Humphrey 1989), p.3.
11
(Lenfle & Loch 2010)
12
(Larman & Basili 2003) is an historical account.
13
(Gilb 1977)
14
(Booch 1996)
15
(Boehm 1988)
16
(Beck 1999)

The failure of process control for software is understandable. Two assumptions are
required for applying formal process control: consistency of artifact, and normality of
variation.17 Because artifacts in software development are not consistent, and variation
does not follow the normal distribution, software development is therefore not a
classically controllable process within error bounds.18 Processes with these characteristics
are considered to require “empirical” control, based on frequent measurement and
correction. Scrum co-founder Ken Schwaber cites the influence of empirical process
control in his rejection of 1990s waterfall methods,19 specifically the work of Ogunnaike
and Ray.20
Agile-aligned authors are therefore critical of attempts to apply statistical process control
to software development, as suggested by the Capability Maturity Model and its
successors. In contrast, critics claim that as software development is a form of R&D,
classical manufacturing improvement approaches are mis-applied.21
Agile methods were enthusiastically adopted by the new generation of Internet startups,
and converged with Cloud computing into a new style of IT delivery.22 According to some
industry surveys Agile is now close to universally adopted among software
organizations.23
Influenced by management thinkers such as Nonaka,24 Lean product management is
becoming an important delivery paradigm,25 changing and in some cases diminishing the
role of project management.26 Philosophies of design thinking and Lean UX drive the
industry desire for an hypothesis-driven, empirical, customer-focused experimental
approach to product understanding and development.27 Fast product feedback is
recognized as an essential success factor for value realization,28 and thus the tempo of
digital product deployment has increased dramatically.29 Since digital products are
increasingly delivered as services, operational discipline is becoming a primary concern,30
sometimes associated with the term “Site Reliability Engineering” (a discipline
associated with large Web properties such as Facebook and Google).31
17
(Wheeler 2000)
18
Credit to Murray Cantor, Ph.D., Aptage CTO, for assistance with this section.
19
(Schwaber 2002)
20
(Ogunnaike & Ray 1994). Schwaber (2002, ch 2) describes meeting a group of industrial control theorists working at Dupont at the same time
he was, including Dr. Ogunnaike himself. These theorists encouraged him to apply empirical methods of frequent assessment and correction to
the software process, as it inherently could not be controlled in the same way as manufacturing processes.
21
(Binder 1997), also (Reinertsen 2009): “[Six Sigma is] commendable in the repetitive world of manufacturing [but] very dangerous in product
development…” (p. 218).
22
see (Limoncelli et al. 2014), appendix B for this history
23
(HP 2015)
24
(Takeuchi & Nonaka 1986), 1986
25
(Cantor 2011; Poppendieck & Poppendieck 2003; Reinertsen 2009; Schwaber & Sutherland 2012).
26
(Cagan 2008), ch. 3; (Memon 2014; Cobb 2015; Koskela Gregory 2002; Apke 2015).
27
(Jeff Gothelf & Josh Seiden 2013; Lockwood 2010; Sussna 2015)
28
(Reinertsen 1997; Ries 2011; Blank 2013)
29
Allspaw & Hammond, 2009; Humble & Farley, 2011
30
(Allspaw & Robbins 2010; Limoncelli et al. 2014)
31
(Limoncelli 2012; Beyer et al. 2016)

This new style relies on incremental and iterative test driven development,32 and ongoing
refactoring to prevent and correct technical debt.33 It calls for continuous integration of
code,34 infrastructure automation (also known as “infrastructure as code”)35 and automated
build management,36 dynamic capacity on demand,37 and loosely-coupled, asynchronous
“microservice” systems architected in keeping with the CAP theorem.38 Strict adherence
to common API standards (REST, JSON) ensures service interoperability.39
Specialized functional (“silo”) segregation is replaced by cross-functional collaboration,
enabled by (at firms like Amazon and Spotify) “two-pizza team” product-centric cells of
no more than 8 participants.40
Employers now seek “T-shaped” professionals (deep-skilled in one area, shallow-skilled
in many areas) who have flexible roles on fast-moving digital product teams.41 The
importance of high-trust, psychologically safe teams is gaining increasing attention due to
the quantifiable impact culture can have on software development and delivery.42
Web- and Cloud-scale engineers point to the influence of complex systems research.43
Systems thinking is a guiding influence.44 The Cynefin framework by Dr. Dave Snowden
has gained attention in the Agile world as a foundational sense-making artifact explaining
the overall relationships between simple, complicated, complex, and chaotic modes of
operation.45
Security remains an important element in the overall development process, but traditional
security processes can interfere with the pace and dynamic nature of agile and DevOps
techniques. Scaling and integrating security requires new approaches including building
requirements into the process, adaptable testing, and integrating with developer
education/awareness and tools.46
32
(Beck 2003; Freeman & Pryce 2010)
33
(Fowler & Beck 1999; Ambler & Sadalage 2006; Brown 1998)
34
(Duvall et al. 2007)
35
(Morris 2016)
36
(Duvall et al. 2007; Humble & Farley 2011; Shortland & Lei 2012; Bass et al. 2015)
37
(Allspaw & Robbins 2010; Limoncelli et al. 2014)
38
(Brewer & Fox 1999; Newman 2015)
39
(Hernandez 2012)
40
(Kniberg & Ivarsson 2012; Choi 2014)
41
see (Reinertsen 2009) “The Principle of T-Shaped Resources: Develop people who are deep in one area and broad in many”
42
(Rozovsky 2015)
43
(Allspaw & Robbins 2010), Chapter 7
44
e.g. as discussed in (Larman & Vodde 2009) Chapter 2, “Systems Thinking.”
45
(Snowden & Boone 2007)
46
(Securosis 2014; Bartsch 2011)
Renewing the IT Curriculum P a g e | 10

A recent watershed moment was the emergency drafting of Google site reliability
engineers (SREs) who used Agile/DevOps methodologies to intervene in the failing
Healthcare.gov initiative, which was being run according to the accepted “best practice”
wisdom. This remarkably successful turnaround raised the visibility of these new IT
practices worldwide47 and has resulted in their advocacy even in the US federal
government,48 long a bastion of the legacy practices. Two new agencies have been created
to assist in their adoption.49
Research firm Gartner Group notes that “DevOps has been by far the most popular search
term on http://gartner.com related to IT operations in 2015.”50 The annual State of
DevOps report presents extraordinary, order of magnitude claims in terms of IT delivery
and stability for organizations with mature Agile and DevOps practices:51
 200x more frequent deployments
 24x better Mean Time to Recovery
 2,555x shorter lead times
 3x lower change failure rate
This same research also indicated that some traditional IT “best practices” are
questionable. For example, the State of DevOps research found that having Change
Approval Boards (a staple of the IT Infrastructure Library process framework) approve
production changes does not result in the intended benefit of stability, while delaying
delivery significantly.52
The same research shows that “practices that make up continuous delivery — deployment
automation and automated testing, continuous integration, and version control for all
production artifacts — have a significant predictive relationship to deployment pain, IT
performance and change fail[ure] rate. In turn, IT performance predicts organizational
performance, as measured by productivity, market share and profitability.”53
Defining “IT,” “Agile,” “DevOps,” and “digital”

Educators and scholars operate in society and must deal with ambiguous terminology.
The following definitions will be used here.
Information Technology (IT)

Information Technology has a problematic history of definition. As noted in the 2005
ACM "Computing Curricula 2005: The Overview Report":
47
(Brill 2014)
48
(Schwartz 2014)
49
(Balter 2015; Lee 2016)
50
(Williams & Holub 2016)
51
(Forsgren et al. 2016)
52
(Labs 2014), p.18.
53
(Puppet Labs 2015), p.13.

“Information technology is a label that has two meanings. In
the broadest sense, the term information technology is often used to refer
to all of computing. In academia, it refers to undergraduate degree
programs that prepare students to meet the computer technology
needs of business, government, healthcare, schools, and other
kinds of organizations. In some nations, other names are used for
such degree programs.”54
In the industrial context, “IT” is usually associated with a centralized organizational
function for managing information technology, a “back-office” concern secondary to the
business. However, an important theme of digital transformation is elevating IT’s role to
include market-facing responsibilities and increasingly close and ongoing interactions
with business partners. Our use of the term IT reflects this evolution.
Agile and the Agile ecosystem

The term “Agile” is also contentious. Properly and strictly speaking, it refers to the
principles espoused by a group of leading software engineers who met in 2001 and
produced the “Agile Manifesto.”55 Its well-known four principles consist of:
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
As the quote in this paper’s introduction from Rigby, Sutherland, and Takeuchi suggests,
Agile principles have proven remarkably influential, such that they have been claimed as
inspiration in various domains beyond the original interests of the Agile Manifesto
authors. There are books on Agile Hiring,56 Agile Contracts,57 and Agile Organization
Design.58
Agile influences are seen “upstream”, e.g. in practices such as Lean UX and Lean
Startup.59 “Downstream” influence is seen in the current “DevOps” movement.60
54
(Shackelford et al. 2005), p. 14.
55
(Agile Alliance 2001)
56
(Sean Landis 2011)
57
(Opelt et al. 2013)
58
(Narayam 2015)
59
(Ries 2011)
60
(Ries 2011; Humble & Farley 2011; Kim et al. 2013)

Conversely, various Agile thinkers and practitioners have brought in influences from
fields other than software engineering. Lean thinking has long been seen as compatible
with Agile principles.61 The Theory of Constraints,62 an important Lean-related set of
ideas, is reflected in works such as Gene Kim’s The Phoenix Project and Larman &
Vodde’s Scaling Lean and Agile Development.63 Product management approaches and
insights64 have influenced notable current practices such as Scrum65, Kanban66, and the
Scaled Agile Framework.67 Other influences include systems theory68 and even control
theory.69
Because of this breadth, we strive to use terms such as “Agile-influenced” or “Agile
movement” when referring to concepts or practices not directly represented in the
original Agile manifesto. We do capitalize the word “Agile” as we recognize the central
influence of the Agile Manifesto.
DevOps
Wikipedia defines DevOps thus:
“DevOps (a clipped compound of development and operations) is a
culture, movement or practice that emphasizes the collaboration and
communication of both software developers and other information-
technology (IT) professionals while automating the process of software
delivery and infrastructure changes. It aims at establishing a culture and
environment where building, testing, and releasing software, can happen
rapidly, frequently, and more reliably.”70
Another important definition comes from Jez Humble, author of Continuous Delivery:
Devops is about aligning the incentives of everybody involved in
delivering software, with a particular emphasis on developers, testers,
and operations personnel. A fundamental assumption of devops is that
achieving both frequent, reliable deployments and a stable production
environment is not a zero-sum game. Devops is an approach to fixing the
first three problems listed above through culture, automation,
measurement, and sharing. 71
61
(Poppendieck & Poppendieck 2007; Poppendieck & Poppendieck 2003)
62
(Goldratt & Cox 2004)
63
(Kim et al. 2013; Larman & Vodde 2009)
64
(Takeuchi & Nonaka 1986; Reinertsen 2009; Blank 2013; Cagan 2008)
65
(Schwaber & Sutherland 2012; Rubin 2012)
66
(Anderson 2010)
67
(Leffingwell 2011)
68
seen e.g. in (Larman & Vodde 2009)
69
as in (Kennaley 2010)
70
(Wikipedia 2016)
71
(Humble & Molesky 2011), “culture automation measurement & sharing” (CAMS) attributed to a John Willis web reference that appears
defunct. CAMS is frequently identified as a DevOps core set of principles.

The Agile origins of DevOps are clear; a watershed moment in its evolution was a
discussion on “Agile Infrastructure” between two industry professionals.72 If we accept
that the term “Agile” is properly applied primarily to software development, then
“DevOps” is the broader term as it encompasses upstream and downstream concerns,
including culture. However, DevOps is less concerned with topics such as hiring and
financial management, IT governance, field services, and so forth. It is not a substitute for
the entire spectrum of IS/IT concerns.
Digital
Finally, the term “digital” (as in “digital transformation”) has a great deal of currency. 73
This paper suggests a pragmatic definition of “digital transformation” as
“the increasing IT component of all that we know, use, experience, and consume.”
(“IT” being defined in the broadest sense of applied computing.) It is our contention that
digital management necessarily implies DevOps and the Agile ecosystem, but is itself a
broader concept. Where the ecosystem’s boundaries become unclear, we switch to the
word “digital.” In particular, the concerns and practices of site reliability engineering and
web-scale IT,74 while compatible, have different motivations and historical origins as
compared to Agile and DevOps. Other frequently mentioned “digital” concerns include
Internet of Things, social media, big data, analytics, mobility, and Cloud.
Competency areas
We suggest the following competency areas for digital curricula:
 Dynamic infrastructure and operations
 Continuous delivery
 Product management
 Resource and execution management
 Organization and culture
An important concern for educators is topic volatility. Technological trends are
notoriously ephemeral. This report is a response to long-term trends in the industrial
application of computing – changes now in evidence for nearly a decade, and that we
believe will persist.
These five competency areas represent what we believe are the most critical topics to
consider for next-generation curricula. Table 1 presents each area’s value proposition.
72
(Davis & Daniels 2016), ch. 3.
73
(Westerman et al. 2011; Westerman et al. 2012)
74
(Allspaw & Robbins 2010; Limoncelli et al. 2014; Beyer et al. 2016).

Table 1. Competency area value propositions
No. Competency area Value proposition
CA1 Dynamic infrastructure and From physical, hand-configured infrastructure to virtual,
operations software-defined infrastructure
CA2 Continuous delivery From “code complete” to “full lifecycle”; from batch
handoffs to ongoing digital value production
CA3 Product management From deliverable-focused project management to outcome-
focused product management
CA4 Resource and execution From execution models resulting in overburden,
management multitasking and poor IT delivery, to concern for value,
flow, and work in process
CA5 Organization and culture From inattention to culture, to recognition of culture’s
central role in digital product delivery effectiveness
The “Dynamic infrastructure and operations” competency area is the most tightly coupled
to the technology refresh cycle. Even that area has emergent practices that we believe
will persist for the long term, such as infrastructure as code and the principles of site
reliability engineering. The other four topic areas are in general representative of
principles and practices that are applicable to most technologies, and of longer cycle
transitions.
Note that this report is not intended to provide full coverage of an entire computing-
related discipline such as Information Systems, Software Engineering, or Information
Technology (as an academic program).
Approach to learning objectives

In general, the learning objectives progress from simple summary/awareness to both
complex applied objectives (suitable for informing labs) and advanced conceptual topics.
In this report version, most but not all learning objectives have supporting references. In
particular, the earlier sections of the first competency area have some gaps in references,
as fulfilling them depends on selecting particular technologies.
Reference note: We switch here from footnotes to inline references.

CA 1 – Dynamic infrastructure and operations
Infrastructure management practices are undergoing a generational shift away from
functional devices managed by console and command line, to more reliable, scalable, and
consistent approaches such as “infrastructure as code” and software-defined networking.
The transition to Cloud providers as a primary infrastructure strategy for many companies
presents new educational opportunities in sourcing and infrastructure and applications
design.
Some overall observations:
 Studying virtualization concepts is an important precursor to covering Cloud.
 Infrastructure as code, with accompanying attention to source control, applies to
public Cloud, private cloud, hybrid Cloud, and traditional static hosting.
 Cloud does not require infrastructure as code (IaC), but using IaC to teach Cloud
is recommended, as Cloud services are at their most powerful when using IaC to
leverage techniques such as auto-provisioning and auto-scaling.
 The authors of this report regularly hear industry concern that current computing
programs have insufficient focus on the centrality of version control as an
enabling industrial discipline, and the challenges of implementing, managing,
and optimizing it. This may seem an obscure technical practice to some, but in the
words of Andrew Clay Shafer, “version control is the foundation of every other
Agile technical practice” (Allspaw & Robbins 2010, chap.16). The State of
DevOps report provides statistical evidence for the importance of version control
(Puppet Labs 2015, p.13). Version control is deliberately included as an
infrastructure topic rather than a development topic, as it is now required across
the full digital stack.
 Highly scaled digital product companies such as Google and Facebook have
pioneered a new discipline of “Site Reliability Engineering” (SRE). SRE
according to Wikipedia is “a discipline that incorporates aspects of software
engineering and applies that to operations whose goals are to create ultra-scalable
and highly reliable software systems.” It is developing into a deep, empirically
validated, and rigorous set of industrial practices (Beyer et al. 2016).
 Continually stressing and testing production systems is increasingly an accepted
part of reliability engineering. Chaos Monkey and the rest of the Netflix Simian
Army (Izrailevsky & Tseitlin 2011) is an important example of such practices and
is freely available to instructors, at the cost of some effort to configure.
Faculty interested in this competency area in particular are encouraged to review the
section on Digital Labs and Simulations.

Virtualization
Note on sources; there are many online tutorials and sources for virtualization, Cloud,
and version control fundamentals, which are some of the faster moving technical areas of
the material covered in this report. Meeting many of these learning objectives require
choice of particular technology (e.g. VirtualBox, KVM, VMWare, Vagrant, Docker, git,
Mercurial, etc) or particular Cloud provider (Amazon, Azure, Google, etc.). The
instructor should refer to those products’ documentation as appropriate.
Virtualization Basics
Portnoy (2016) is generally applicable.
 Explain the basic purpose of virtualization (concept, hypervisor, etc.)
 Compare and contrast the two types of hypervisors-bare metal versus operating
system resident
 Identify current generation Type 1 and 2 hypervisors by name
 Explain the economic rationale for virtualization (e.g. increased capacity
efficiency)
 Install Type 1 or Type 2 hypervisor
 Install, manage, clone, and decommission a virtual machine
Virtualization Operations
Portnoy (2016) is generally applicable.
 Configure basic settings for a virtual machine (RAM, vCPU, VNIC, storage)
 Change IP network for virtual networks
 Demonstrate working with a virtual machine (e.g. migrating to another host)
 Deploy a virtual machine programmatically using Vagrant
Virtualization Security
 Identify common built-in hypervisor-level security options
 Describe unique security requirements of VM guests
 Identify common add-on security options (e.g. hypervisor-level firewall, AV,
NSX, etc.)
Virtualization Engineering
Advanced challenges suitable for capstone projects. See section “Digital labs and
simulations.”
 Design/build/demonstrate a server-client network with virtualization
 Design/build virtual networking infrastructure
 Design/build high availability VM infrastructure
 Design/build VM storage infrastructure

 Design/build/demonstrate various other VM devices-network monitoring,
firewall, intrusion prevention device, intrusion detection device, software defined
networking device
Containerization
Matthias (2015) is applicable for the Docker platform in particular.
 Describe the purpose of containerization
 Describe the differences between containers and virtual machines
 Describe the benefits of using containers versus virtual machines (e.g.
environmental consistency)
 Describe the basic components of a container solution
 Install, configure, manage, and decommission a container (e.g. LXC Linux or
Docker)
 Describe the purpose and value of container management and orchestration
tooling (Mesos, Swarm, Fleet, Kubernetes, Docker Compose)
 Demonstrate the use of container orchestration tooling in a virtual laboratory or
simulation.
 Describe the potential security concerns associated with containers and the
container daemon
Cloud
Cloud Basics
NIST (2011), Ruparelia (2016), and Bond (2015) are generally applicable.
 Explain the basic purpose and characteristics of the cloud
 Describe how virtualization is being used in the cloud
 Describe the basic characteristics of cloud services (elasticity, scalability, multi-
tenancy, etc.)
 Describe the basic service models of IAAS, PAAS, and SAAS
 Describe the basic deployment models (public, private, community, and hybrid)
 Identify current generation cloud providers by name
 Describe the security considerations when moving or processing internal data at a
cloud service provider (Ruparelia 2016; Bond 2015).
 Identify and describe practices and patterns for Cloud-based architecture
(Amazon Web Services 2016a)
 Identify and describe the continuum of Cloud compute from virtual machines to
containers (Amazon ECS, Google Container Engine, Azure Container Service) to
serverless (e.g. Amazon Lambda, Azure Functions)
 Identify and describe Cloud-based storage and data management products
 Identify and describe Cloud-based networking products

 Describe Cloud security practices and patterns, such as pervasive encryption and
key management
 Describe the shared responsibility model for Cloud security (provider responsible
for lower layers, customer responsible for higher layers) (Amazon Web Services
2016b)
 Describe the security considerations of the IAAS, PAAS, and SAAS service
models
Cloud Practices
 Set up account in a cloud
 Install server in a cloud
 Set up networking, access credentials, and security policies in cloud.
 Install software on cloud-based server
 Configure multiple servers to interact with each other in the cloud, according to
typical architecture patterns
Cloud Sourcing
 Describe the economic value proposition of Cloud infrastructure (Ruparelia 2016)
 Compare and contrast a Cloud sourcing approach with a traditional in-house data
center
 Describe and discuss the challenges associated with moving traditionally hosted
applications to Cloud environments (Bond 2015)
 Discuss the legal and contractual concerns of Cloud sourcing (Millard 2013)
Software Defined Infrastructure
Version Control Basics

Morris (2016) and Duvall (2007) are generally applicable.
 Describe the purpose of version control (VC)
 Obtain version control account
 Install version control software
 Use version control software to initialize, create, commit, modify a repository
 Describe the basic components of a version control system
 Identify the differences between source and package management
 Describe the differences between centralized and decentralized version control
systems
 Describe the basic version control process
 Describe and demonstrate familiarity with forks, branches, tags, and merges
 Demonstrate familiarity with major Cloud-based version control systems
(GitHub, BitBucket)
 Describe the deployment process (Dev > QA > Production)

 Use VC software to download a repository
 Use VC software to upload a repository change
 Describe security components within version control (change control, automated
testing, etc.)
 Configure appropriate user controls for a repository
Automation & configuration

Morris (2016) is generally applicable.
 Identify purpose and components of an automated configuration manager
 Compare and contrast imperative versus declarative (policy-based) approaches to
configuration management (Burgess 2004a; Burgess 2004b)
 Describe the basic automation process
 Install a configuration manager in a lab environment
 Configure various security roles within the automation and/or configuration
manager
 Configure automation and configuration manager security features
 Configure a base virtual machine using Infrastructure as Code techniques,
including shell scripting and/or policy-driven configuration management to
configure the VM (examples: Chef (Taylor & Vargo 2015), Puppet, Ansible, Salt
Stack)
 Install software on a virtual machine using Infrastructure as Code techniques
 Define immutability and explain why it is important (Wiggins 2012)
 Describe the purpose and use of an API
 Invoke a basic API call (with parameters provided)
 Configure multiple virtual machines or containers via Infrastructure as Code to
instantiate as a functioning distributed system within a contained sandbox
network.
 Upload infrastructure artifacts for collaboration into version control system
Software-Defined Networking
Goransson and Black (2014) is generally applicable.
 Define concept of Software-Defined Networking
 Describe the main SDN architecture (e.g. controller, API, layers/plane
(management, control, and data/forwarding))
 Describe network function virtualization
 Describe the potential security concerns of the various SDN components
 Describe the potential security advantages when using SDN
 Use the main SDN components to build a small network

Site reliability engineering
Staffing concerns
 Describe Google’s approach to staffing Site Reliability Engineering teams (Beyer
et al. 2016).
 Describe Google’s approach to managing the workloads for Site Reliability
Engineering teams (Beyer et al. 2016).
Availability and risk

 Describe how cost scales (exponentially) with increasing reliability (Beyer et al.
2016)
 Describe the opportunity cost of availability investments (Beyer et al. 2016)
 Describe why Google implemented an "error budget" (Limoncelli et al. 2014;
Beyer et al. 2016)
Automation and operations

 Identify Google’s definition of "toil" (Beyer et al. 2016)
 Identify the basic elements of a monitoring system, such as event, alert, log,
dashboard (Beyer et al. 2016)
 Configure a basic monitoring capability using infrastructure as code techniques
 Implement monitoring for program/feature availability (e.g. specific services)
Designing for scale and failure

(Nygard 2007; Hamilton 2007) are generally applicable.
 Identify importance of, approaches and practices for designing for failure
(Amazon Web Services 2016a)
 Identify the principles of the Reactive Manifesto (Bonér et al. 2014)
 Identify types and methods of load balancing (Limoncelli et al. 2014; Beyer et al.
2016)
 Describe the role of a Content Delivery Network (Allspaw & Robbins 2010;
Limoncelli et al. 2014)
 Identify the CAP theorem and its tradeoffs (Brewer & Fox 1999; Limoncelli et al.
2014)
 Identify and define the axes (horizontal, functional, lookup) of the AKF Scaling
Cube (Abbott, Martin L.; Fisher 2015; Limoncelli et al. 2014)
 Describe approaches (rolling, canary, feature toggles, etc) for upgrading large-
scale, live services (Limoncelli et al. 2014)
 Demonstrate scaling techniques using software-defined laboratory clusters
 Demonstrate CAP theorem tradeoffs practically in a lab setting, using techniques
such as sharding and/or read replicas

Operational disruption
 Identify key characteristics of complex systems and how they fail (Allspaw 2012;
Allspaw & Robbins 2010)
 Identify elements of an incident management process (Beyer et al. 2016) (Beyer
2016 ch. 14)
 Describe staffing approaches for on-call operations (Limoncelli et al. 2014)
 Identify and define the roles and responsibilities of the incident command post
(Beyer et al. 2016; Limoncelli et al. 2014)
Causal analysis, mitigation, and remediation

 Identify elements of postmortem processes (Allspaw & Robbins 2010; Beyer et
al. 2016)
 Compare and contrast causal analysis approaches focusing on human failure and
accountability, versus blameless approaches (Dekker 2014)
 Describe the motivation behind Netflix’s Chaos Monkey tool (Izrailevsky &
Tseitlin 2011).
 Implement Chaos Monkey or other Simian Army tools in a virtual lab
environment.
Security and privacy

 Identify and describe privacy concerns (particularly with cloud or external data)
 Identify and describe current regulatory concerns affecting Cloud usage (HIPAA,
European data privacy)
 Described the Amazon shared-responsibility security model (Amazon Web
Services 2016b)

CA 2 – Continuous delivery
Computing is increasingly delivered as an operational service upgraded on a frequent
basis. This in turn requires curricular attention to software’s deployment and operations,
subsequent to “code complete,” testing, and packaging.
In current curricula, there is inadequate support for later-stage software lifecycle concerns
(Release, Deploy, Operate). Courses still are often scoped by classic waterfall project
phases (requirements management, systems analysis, construction practices, testing/QA).
Despite its growing reach and influence across the lifecycle, Agile is still often covered
merely as a subset of project management.
Further attention to testing and test-driven development is needed; like version control, it
should be integrated into all programming classes. This includes full lifecycle testing,
including integration testing, acceptance testing, installation testing, performance testing,
security testing, and so forth.
This section is focused on the end to end delivery pipeline, its components, and its
enabling of fast feedback. The concept of “release engineering” is sometimes associated
with this scope. The concept of “DevOps” however is broader, encompassing culture and
to some extent product management (Forsgren et al. 2016).
Agile methods, in the pure sense of software development, have been receiving academic
attention for some time and this report does not claim to fully cover all aspects of Agile
per se. Again, the learning objectives are examples for reference and no claim is made
that they are exhaustive.
Agile and related software development practices
Core Agile practices

 Identify and describe the importance of the Agile Manifesto (Agile Alliance
2001)
 Identify the core practices of eXtreme Programming (Beck 1999)
 Define and distinguish iterative and incremental development (Agile Alliance
2016)
 Describe the importance of small, cross-functional, multi-disciplinary teams
(business users, designers, developers, testers and operations analysts) in Agile
methods (Cohn 2010, pp.177–182).
 Define and distinguish Behavior Driven Development, Acceptance Test Driven
Development, and Test Driven Development (Agile Alliance 2016)
 Engage in pair programming to develop a software application (or a web page,
mobile app, etc.)
 Identify and describe Agile influences in domains beyond software development
(Opelt et al. 2013; Sean Landis 2011; Rigby et al. 2016; Arbogast et al. 2012;
DeLuccia et al. 2015)

Describing system intent
See also “Product Management” competency area for details on product management
 Identify and describe various approaches to describing system intent (traditional
functional and nonfunctional requirements, use cases, user stories, behavior-
driven design)
 Discuss alternatives to effort estimation, such as planning poker (Rubin 2012,
chap.7)
 Compare and contrast traditional requirements with user stories (Cockburn 2001)
 Construct system requirements using user stories and experiences
Application design
 Describe the fundamental architectural components of digital products
(Limoncelli et al. 2014)
 Define refactoring, and explain the importance of test-driven development
(Freeman & Pryce 2010) in supporting refactoring (Fowler & Beck 1999)
 Discuss design & development considerations for cloud vs data center (Allspaw
& Robbins 2010, chap.2)
 Identify the 12-factor approach to application architecture and development
(Wiggins 2012)
Continuous Delivery pipelines
Understanding the end to end digital pipeline

 Describe benefits (improved product discovery through faster feedback, better
stability) to an organization in implementing continuous delivery (Allspaw &
Robbins 2010; Humble & Farley 2011; Bass et al. 2015; Kim et al. 2016)
 Describe challenges to an organization in implementing continuous delivery
(Humble & Farley 2011)
 List and describe the "Three Ways of DevOps" (Flow, Feedback, Continual
Learning) according to Kim et al (Kim et al. 2013; Kim et al. 2016)
 Identify the core components to a Continuous Delivery pipeline (Farcic 2016;
Bass et al. 2015; Kim et al. 2016)
 Use an architectural approach to describe continuous delivery approaches and
concerns (Open Group IT4IT Forum Agile Workgroup 2016)
 Describe various ways in which an automated DevOps toolchain supports goals of
auditability (DeLuccia et al. 2015)
Managing the development environment

 Identify the importance of development environment consistency with
“downstream” environments (Humble & Farley 2011, pp.5–12)
 Describe how infrastructure as code and virtualization can support a standard
desktop development environment (Hashimoto 2013)

Understanding source control and continuous integration
 Track the evolution of a software module over time using version control (Chacon
2009; Chacon 2011; Duvall et al. 2007)
 Identify various source code branching and merging strategies (release, feature,
team, no branching) and their pros and cons (Humble & Farley 2011, chap.14;
Forsgren et al. 2016 – higher performing teams integrate more frequently).
Automated testing
 Explain the practice of test driven development (Beck 2003; Freeman & Pryce
2010)
 Explain the benefits and various applications of automated testing (e.g. Gruver et
al. 2013 – discussion of test automation in HP firmware)
 Compare and contrast various automated testing and build tools (Acharya 2014;
Holzner 2009; Sonatype 2009)
 Implement one or more automated unit tests to validate functionality of code
(Kaczanowski 2013; Freeman & Pryce 2010; Acharya 2014).
 Implement an automated build (language-specific) of a given source module
(Holzner 2009; Sonatype 2009)
Build management
 Install a build choreography manager and demonstrate using it to control
heterogeneous builds (Smart 2011)
 Integrate a source repository and build choreography manager (Smart 2011)
Code analysis and peer review

 Use tools to calculate test code coverage
 Use automation tools (e.g. static analyzers) to review code for best practices and
common errors
 Describe peer review workflow (DeLuccia et al. 2015)
 Use pull requests in a version control software system as a basis for reviewing,
critiquing, and merging tested code.
 Integrate code coverage, static analysis, and peer review with automated build
choreography
 Critique software documentation and produce feedback on the documentation for
the author
Package management
 Describe the functions of a package manager (Humble & Farley 2011, chap.6)
 Describe the role and value of a package manager with large multi-team
codebases (saves build time, ensures that teams are working with other teams
"blessed" versions)

 Describe the risks of upstream package dependencies on the software ecosystem
(Williams 2016)
 Describe the role and value of a package manager in controlling external
resources (e.g. open source libraries - ensures availability and consistency)
 Describe interaction of package manager with build manager and deployment
toolsets
 Install and utilize a package manager as part of a lab-based DevOps toolchain
Deployment and integration testing

 Identify and describe the testing pyramid pattern and snowcone antipattern – unit
test automation should be the broadest foundation (Newman 2015; Morris 2016)
 Demonstrate user interface testing in a lab setting, e.g. using a tool such as
Selenium.
 Describe architecture alternatives for deployment into pre-production and
production environments (Allspaw & Robbins 2010, chaps.4, 10; Humble &
Farley 2011)
 Explain A/B testing and how it is implemented (Allspaw & Robbins 2010,
chap.10)
 Explain testing for reliability vs testing for features
 Describe the architecture required for performance or load testing
 Implement a load testing environment (e.g. with JMeter)

CA 3 – Product management
There is a need for updated coverage of the organizational context driving digital
investments. Product management is a critical aspect ot this organizational context.
Strictly speaking in traditional curricula, Product Marketing is found in business school
programs, and is distant – in the curricular sense – from technical disciplines such as
Computer Science. Such disciplinary separation is similar to the “business/IT” divide
often viewed as dysfunctional in enterprises.
Ries’ Lean Startup (2011) is an influential practice both in startups and in enterprises,
with its emphasis on tight, “fast feedback” experimentation loops to discover product
value. This requirement for fast product feedback is resulting in the breaking down of the
“business/IT” barrier in modern digital organization. In industry, successful product
managers maintain close and ongoing contact with their technical partners, often in
the context of cross-functional product teams. Correspondingly, according to recent
research from startup incubator Y-Combinator, product-oriented programmers are the
most desired by startups:
…The “Product Programmer” and “Technical Programmer” profiles
are identical, except one is motivated by product design, and the other by
solving hard programming problems. There is almost twice as much
demand for the product programmer among our companies. And the
“Academic Programmer” (hard-problem focused, but without the
experience) has half again the demand (Bartram 2015).
A related workforce topic is the creation of tens of thousands of “product owners” via the
popular Scrum methodology. The growth of this role (primarily driven at this time by
industry training) suggests a need for more guidance on product management in current
computing-related college curricula. Anecdotally, these Scrum “product owners” rarely
come from a Marketing background, and more often are business analysts, senior
developers, project managers, and so forth.
The implications for traditional curricula boundaries are interesting. Educators are
encouraged to consider and experiment with cross-department collaboration. Existing
Requirements Management and Usability courses can provide an important foundation.
Digital product management origins and influences
Understanding digital product management origins

 Describe the "consumerization" of digital products and how this is different from
"back-office" industrial IT, as a function
 Identify the digitally-enabled aspects of daily life experiences
 Describe Amazon’s digital product approaches, such as always designing on the
assumption the service may be marketed externally, and keeping team size small
(Bass et al. 2015)

 Characterize the digital practices of the Silicon Valley startup economy (Google,
Facebook, Apple, etc.) and how they differ from older organizational models with
centralized IT functions
 Compare and contrast digital product management and traditional enterprise IT
project management (Narayam 2015)
 Describe the concept of "outcome" as distinguished from "deliverable" (Jeff
Gothelf & Josh Seiden 2013, “outcomes not output” principle).
Understanding role of marketing and brand management

 Identify key concerns of traditional marketing, brand, and product management
(Kahn & Product Development & Management Association. 2005)
 Identify Product Development and Marketing Association (PDMA) as
professional organization
 Describe Cagan’s views on product marketing versus product management
(Cagan 2008).
Understanding Lean product management

 Describe concepts of Lean startup (Ries 2011)
 Describe Gothelf’s foundations of Lean UX (Jeff Gothelf & Josh Seiden 2013)
o Design thinking (Lockwood 2010)
o Agile
o Lean Startup
 Describe Reinertsen’s concepts/concerns of Lean product management
(Reinertsen 1997; Reinertsen 2009)
o Economic foundation
o Fast feedback
o Managing queues
o Limiting invisible work in process
o Batch size importance
 Define and distinguish lean production (Lean in the manufacturing context) from
lean product development (Reinertsen 2009)
Organizing and managing the product team
Understanding roles in product management

 Identify the Patton triad (Patton 2014)
o "Business person,", e.g., who asks, "is it valuable?"
o "UX specialist or interaction designer,", e.g., who asks, "is it usable?"
o "Technologist,", e.g., who asks, "is it feasible?"
 Analyze the relationships of these three roles and potential issues in those
relationships

 Compare and contrast different views on the relative roles of product manager and
product owner For example, (Cagan 2008) argues they should be the same, and
(Rubin 2012) discusses various options.
Employing Scrum and the "product owner"

 Describe the principles and practices of Scrum at an overview level (Schwaber &
Sutherland 2012; Rubin 2012; Cohn 2010)
 Describe the current scope, role, and importance of Scrum as a widely adopted
development methodology
 Identify the origins of Scrum in product management and applied control theory
(Schwaber 2002)
 Demonstrate familiarity with Scrum roles (Schwaber & Sutherland 2012; Rubin
2012) including
o Product owner
o Scrum master
o Team member
 Demonstrate in a classroom team setting the ability to undertake typical Scrum
activities, including:
o Standups
o Actual design & development work
o Retrospectives
o Demonstrations
See also Competency Area 4 for execution aspects of Scrum.
Managing the digital product lifecycle
Discovering product ideas

 Describe concept of product/market fit
 Describe the basic practices of market research
 Recognize and identify appropriate uses for Osterwalder’s Business Model
Canvas (Osterwalder & Pigneur 2010)
 Describe key principles of Jeff Gothelf’s Lean UX approach (Jeff Gothelf & Josh
Seiden 2013)
 Describe importance and approach of interviews in product discovery
 List and describe techniques for capturing, analyzing, and communicating product
characteristics
o Traditional requirements
o User story mapping (Patton 2014)
o Impact mapping (Adzic 2012)
 List, describe, and compare and contrast various methods & techniques for
validating product ideas (e.g. Ries 2011)

 Describe the limitations of “expert” opinion reliability in product investment
choices. For example, Kohavi et al (2009) found that 1/3 of well-vetted ideas
result in benefits, for 1/3 there is no change in results, and 1/3 detract from
organizational goals.
 Identify and describe the INVEST criteria (Independent, Negotiable, Valuable,
Estimable, Small, Testable)
 Describe Reinertsen’s view of product development as information creation
(Reinertsen 2009, p.86)
 Describe the importance of fast feedback in product development (Reinertsen
2009) (Reinertsen, 2009)
 Identify traditional management practices that inhibit fast feedback, and the
business risk they represent
Delivering digital products

 Demonstrate the use of Kanban to manage development tasks for products and
features (Anderson 2010)
 Demonstrate use of defect/ALM tools for bug & issue reporting and tracking
 Describe microservices as one approach to delivering products (Newman 2015)
 Implement a simple microservice-based product in a lab setting, driven by product
lifecycle practices
 Demonstrate importance of fast feedback for product development, in a lab
simulated setting
Product portfolio management
Managing the investment decision

 Describe traditional project/operations funding and execution
 Describe product portfolio oriented funding and execution
 Compare and contrast project- and product-centric approaches to investment
 Describe the Spotify "DIBB" model (Data, Insight, Belief, Bets) (Kniberg 2016)
 Describe agile portfolio management approaches (e.g. SAFe, Spotify including
DIBB model, others)
 Compare and contrast Agile approaches to architecture: SAFe (Leffingwell et al.
2014), Disciplined Agile Delivery (Ambler & Lines 2012) with established
architecture frameworks such as TOGAF (The Open Group 2015)
 Describe the concept of Cost of Delay and how it may be used in making
investment decisions (Reinertsen 2009, p.31)
Developing large scale product strategy

 Describe origins and use of concept of "epics."
 Define and distinguish "feature teams" from "component teams” (Cohn 2010,
pp.183–188).

 Demonstrate familiarity with principles and practices of business architecture
 Discuss the relationship of strategic planning to Lean Startup learning cycle

CA 4 – Resource and execution management
Project management has long been the reigning delivery paradigm, and is entrenched in
computing curricula guidance and industry frameworks. However, there is increasing use
of cross-functional, team-based, product-centric approaches, some of which no longer
employ the concept of “project” as a primary execution vehicle. Products are outcome
focused, where projects tend to be deliverable focused, irrespective of business result.
There is also concern about execution models that rely on cross-functional processes to
drive results across functionally specialized organizations. Such models tend to introduce
transactional friction and resulting delays in product feedback, and have a tendency to
accept too much “work in process” into the system.
More attention is needed to questions of demand and work management in the context of
complex, stressed digital organizations (e.g. applied queuing theory), especially when
multiple teams are working towards common objectives. It is clear (e.g. from history of
Scrum and the work of Don Reinertsen) that digital product development is increasingly
influenced by techniques and theories from Operations Management and Industrial
Engineering; cross-disciplinary discussions and experiments are recommended. Systems
theory and human factors are additional influences. Because of these developments, we
think educators should take a more objective, critical evaluation of the content,
alternatives to, and real-world outcomes of best-practice industry frameworks (CMMI,
PMBOK, ITIL, etc.), if they are incorporated in curricula.
In the interests of supporting organizations of all sizes, courses should teach a scalable
(down as well as up) approach to IS/IT management concerns such as process
management, governance, and resource management.
This learning objective is in some ways the most abstract and difficult, but we believe it
is essential to evaluating project management and concepts of “software process” in the
curriculum.
See in general Smith & Reinertsen (1998), Reinertsen (1997; 2009), Larman & Vodde
(2009), Cohn (2010, sec.IV), and Davis & Daniels (2016).
Principles for executing and scaling
Cohesion & coupling in team organization (the need for coordination)

 Identify examples of coordination in terms of inter-team efforts towards shared
goals (what do we mean by “coordination”?)
 Identify and describe basic issues of matrix management (product vs functional)
 Identify and describe the Spotify organizational model (Kniberg & Ivarsson 2012)
2012)
 Compare and contrast the Spotify model with traditional functional organization.
(Kniberg & Ivarsson 2012; Kniberg 2016)
 Identify circumstances requiring inter-team coordination when teams are
functionally organized

 Identify circumstances requiring inter-team coordination when teams are product
organized
 Discuss the advantages and limitations of product team encapsulation in reducing
the need for coordination (Cohn 2010, chap.10; Reinertsen 1997, chap.6).
Coordination mechanisms
Reinertsen (2009, chap.7,9) is generally applicable.
 Evaluate, compare and contrast the various mechanisms for inter-team
coordination (Cohn 2010)
 List and describe classes of external obstacles to delivery experienced by product
teams
 Identify typical mechanisms for coordinating work across functionally specialized
teams, and their advantages and disadvantages
 Identify typical mechanisms for coordinating work across product-focused teams,
and their advantages and disadvantages
 Identify the concept of the release train and describe its relationship to product
and project management (Cagan 2008)
 Identify the concept of co-location and its uses, examples, and limitations
 Discuss approaches to optimizing remote working. (Davis & Daniels 2016)
 Identify the concept of cadence and its uses and examples (Reinertsen 2009,
chap.7)
 Identify the concept of synchronization and its uses and examples (Reinertsen
2009; Cohn 2010; Davis & Daniels 2016)
 Identify the origins of the daily standup, its origins, importance, and various
applications
 Prepare for and participate in a stand-up meeting for a software project (see also
Competency Area 3).
 Identify the technique of scrum of scrums and related approaches and their
advantages, disadvantages, and applications (Cohn 2010, pp.340–343) (Cohn
2010 pp 340-343)
 Describe the origins and fundamentals of the Objectives and Key Results (OKR)
performance measurement approach, created by Andy Grove at Intel and used by
Google and other digital firms (Grove 1995)
Demand and Queueing

A dramatization of the negative results of too much work in process in a traditional IT
organization is seen in the Phoenix Project novelization (Kim et al. 2013); queuing
theory is discussed in this context.
 Describe basic principles (from an applied perspective) of queueing theory
(Reinertsen 2009, chap.3)
 Describe the application of queueing theory principles to digital work
management (new features, operational activities)

 Identify typical uses of queues in executing and coordinating work in digital
organizations (ticketing systems, backlogs) and their strengths and limitations
 Identify issues that can be addressed as queuing problems
 Characterize the human ability to service multiple queues simultaneously (the
limitations of human multi-tasking)
 Identify queueing issues with current organizational practices and frameworks
 Use an enterprise architecture approach to identify queues across a digital
delivery pipeline (Open Group IT4IT Forum Agile Workgroup 2016)
Industry practices for execution
Timeboxed approaches
 Describe origins and characteristics of classical project management
 Describe basic practices of Scrum (Rubin 2012)
 Describe Scrum’s origins in product development (Takeuchi & Nonaka 1986)
 Describe Scrum’s origins and approach in terms of process control (Schwaber
2002, chap.2)
Continuous flow approaches

 Identify basic practices of business process management (Rummler & Brache
1995)
 Compare and contrast business process management with project management
 Describe the limitations of seeking/assuming fully-defined processes (Schwaber
2002, chap.2)
 Describe origins and characteristics of Kanban, as a software method (Anderson
2010)
 Compare and contrast timeboxed approaches with continuous flow approaches
(e.g. project management with ongoing product development using ongoing
backlogs)
Product management and execution

 Describe the importance of high-bandwidth team collaboration to product
management (Reinertsen 2009, pp.263–264)
 Define Cost of Delay (Reinertsen 2009)
 Describe why Cost of Delay is critical for product development execution
(Reinertsen 2009, pp.31–32; “if you only quantify one thing, quantify the cost of
delay”)
 Identify delay cost profiles (Arnold 2013)
 Define Weighted Shortest Job First (Reinertsen 2009, pp.192–195)
 Define the concept of beneficial variability and its utility in product development
(Reinertsen 2009, chap.4)

Measuring and managing
 List various approaches to measuring digital organization execution (story points,
burndown charts, releases, continuous flow diagrams, aggregated metrics)
 Describe the importance/significance of the “definition of done” and where it is
relevant. (Rubin 2012, chap.4)
 Identify practices and limitations of statistical process control in digital product
delivery contexts (Schwaber 2002, chap.2; Bollinger C. 1991; Binder 1997;
Reinertsen 2009, chap.4 & throughout; search on “statistics”)
 Identify inherent problems in measuring and managing (gaming the system,
incenting wrong behaviors, etc.)
 Discuss the intersection of measurement and culture (Forsgren 2016)

CA 5 – Organization and culture
Academic computing programs have long discussed social context; for example, ethics
issues are frequently covered in MIS texts such as Laudon & Laudon (2015).
Questions of collaboration and culture are becoming more critical to industry. How are
product-centric teams including business analysts, developers and engineers to be
managed? Recent research by Google indicates that certain cultural factors, including
psychological safety, are common across high-performing teams (Rozovsky 2015). The
importance of culture is also suggested in the annual State of DevOps reports (Forsgren et
al. 2016; Puppet Labs 2015; Labs 2014).
Educating students in cultural aspects is important so that they can recognize if the
employer’s culture supports high performance. It is also critical for computing
professionals seeking to advance along a management track.
Increased attention to cultural factors may also be key to resolving current under-
representation of women and minorities in computing careers. There is opportunity for
cross-disciplinary partnership with organizational theorists, psychologists, and human
factors specialists.
Davis & Daniels (2016) is generally applicable.
Organizing for digital delivery
Evaluating functional versus product-centric organizing

 Identify the importance of team organization to successful delivery of digital
products and services (e.g., Conway’s law) (Narayam 2015; Davis & Daniels
2016)
 List the major types of organizational matrixing (functional silo, cross-functional,
intermediate forms) (Reinertsen 1997, chap.6)
 List the advantages of functional specialization as a basis for organizational
forms, such as cultivating deep expertise and utilizing resources efficiently
(Narayam 2015, chap.5; Smith & Reinertsen 1998, chaps.7–8).
 List the disadvantages of functional specialization on digital and Agile delivery,
such as increased wait times and queueing overload (Narayam 2015, chap.5;
Smith & Reinertsen 1998, chaps.7–8)75
 List the advantages of small product focused-teams as a basis for organizational
forms, such as higher bandwidth interactions / faster feedback & adaptiveness
(Cohn 2010; Bouthillier 2005; Choi 2014; Davis & Daniels 2016)
 Describe a characteristic product team scaling structure, such as seen in SAFe
(services/products/epics/portfolio) (Leffingwell et al. 2014)
75
While Reinertsen and Smith are focusing on the same issue in a manufacturing context, Reinertsen’s thoughts have been influential throughout
the Agile movement.

 Understand various models for change dynamics and team interactions (Narayam
2015, chap.5).
Deciding team persistence

 List advantages & disadvantages of persistent (product-centric) teams
 List advantages & disadvantages of on-demand (temporary project) teams
(Narayam 2015, chap.8)
Sourcing & contracts

 Describe time & materials contracting
 Describe shared-risk contracting
 Describe issues and risks related to subcontracted development work (Davis &
Daniels 2016)
 Discuss current approaches for blending time/materials and shared risk paradigms
(Opelt et al. 2013)
 List issues of managing widely separated teams (Cohn 2010, chap.18)
Adapting and self- organizing

 Identify principles and practices of adaptive team management (e.g. from
Kanban) (Burrows 2015)
 Identify types of team self-organization (dimensions, degrees of freedom)
(Reinertsen 1997, pp.106–108)
 Identify motivations, advantages, and disadvantages to self-organization
Scaling digital and Agile delivery

 Identify common themes in discussions of organizational scaling (Harnish 2014;
Furr & Ahlstrom 2013; Blumberg 2013; Singer 2004)
 Describe how organizations tend to cluster at certain sizes and the concept of the
“valley of death” in scaling from one size to the next (Harnish 2014, chap.2)
 Compare and contrast various Agile and digital scaling approaches and analyses
(Leffingwell et al. 2014; Schwaber 2007; Larman & Vodde 2009; Ambler &
Lines 2012)
Enabling people and understanding culture
Defining culture
 Compare and contrast various definitions of organizational culture, as related to
digital organizations (Allspaw & Robbins 2010; Davis & Daniels 2016;
Limoncelli et al. 2014)
 Recognize warning signs of poor organizational culture, for example cultures that
seek “accountability” for complex systems errors (Dekker 2006); see also US
military research into “toxic command,” (Vergun 2015)

Impact of culture on organizational performance
 Identify the importance of cultural factors to digital product and service delivery
(Puppet Labs 2015; Forsgren et al. 2016; Rozovsky 2015; Davis & Daniels 2016).
 Describe practices that may increase cultural cohesion and effectiveness (shared
responsibility, rotation, cross-functional teams) (Davis & Daniels 2016; Bass et
al. 2015)
 Identify the major characteristics of successful teams (Rozovsky 2015; Lencioni
2002; Pentland et al. 2012)
Cultural typologies
 Identify the Westrum typology (Puppet Labs 2015; Forsgren et al. 2016)
 Describe how current research (State of DevOps) uses the Westrum typology
(Puppet Labs 2015; Forsgren et al. 2016)
 Identify the Westrum aspects correlated with higher team performance (Puppet
Labs 2015; Forsgren et al. 2016)
Organizational transformation
 Identify and describe various perspectives on deliberate organizational cultural
change in terms of feasibility & approach
 Identify and describe approaches to self-organization (Morningstar, Zappos)
 Identify and describe current industry frameworks used for digital organizational
transformation (e.g. SAFe, Spotify, DaD, LeSS, Holacracy, Enterprise Scrum –
will change over time)
 Define and describe the Toyota Kata approach to creating a learning organization
(Rother & Shook 2003)
 Define and identify historical importance of Training within Industry framework
(Graupp & Wrona 2011)
Skills
 Define the concept of T-shaped professional and its importance to organizational
agility (Reinertsen 2009, p.155)
Incentives and accountability

 Identify and describe current research and thinking related to incentives,
accountability, and performance management in digital organizations . (e.g.
(Gneezy et al. 2011)
Behaviors
 Identify techniques for building common ground between teams (Davis & Daniels
2016)
 Describe current research findings on the poor performance of single executive-
driven "gut feel" decisions (Kohavi 2009)

 Identify and describe confirmation bias and how it may affect product decisions
and digital management
 Identify Dweck’s fixed vs. growth mindset and its implications for digital teams
(Davis & Daniels 2016)
 Identify and describe Kahneman’s Type 1 versus Type 2 thinking styles and their
relevance to product decisions and digital management
 List and describe strategies to move organizations from Type 1 to Type 2
thinking.
 Describe the impact of toxic workers on organizational productivity (Housman &
Minor 2015)

Curricula, courses, and pedagogy
Section objectives
In this section, we will consider:
 Challenges with current curricula approaches
 Suggestions for evolving/enhancing commonly seen programs and courses in the
curriculum
 Virtual labs and simulations
Current challenges
This report does not question current educational practices with respect to theory and
principles of computing and information. Students need to understand theoretical and
technical fundamentals such as algorithms and data structures, computing language
design, operating systems and compilers, automata theory, and so forth.
But as stated elsewhere, computing curricula often include contextual material intended
to orient the student to industrial practice. Examples include courses in project
management, requirements management, systems analysis and design, software quality
assurance, and ethics and computing. This report is intended to enhance the educational
approaches to these contextual aspects in light of trends such as digital product
management and continuous delivery.
As mentioned in several locations above, one notable aspect of the Agile and digital
trends is increasing application of insights from other domains. Site reliability
engineering is turning to the human factors literature (e.g. Dekker 2014). Scrum traces its
origins to classical control theory as well as next-generation product development.
Systems theory appears throughout Agile professional literature.
Cross-disciplinary responses are challenging, but we believe increasingly essential for the
student experience. Minimally, course readings should look outside the core canon of
project management, requirements management, and systems architecture (this report
provides many suggestions). Ideally, cross-departmental courses could be considered.
Course-specific recommendations
Creating new courses is difficult. The following are courses typically seen in curricula.
The following are some suggestions for incorporating this report’s themes without
necessarily creating new courses.
Introductory MIS/IT courses

Introductory courses should include relevant material describing the new practices. The
Phoenix Project is an accessible and entertaining novelization for introductory students
covering Agile and DevOps in the context of a traditional IT organization’s struggles
(Kim et al. 2013).

Business analysis and requirements management
Classical business analysis/requirements management courses can be enhanced with Lean
Startup and Agile readings such as Leffingwell (2011), Patton (2014), Gothelf & Seiden
(2013), Ries (2011), and Cockburn (2001).
Project management
Project management classes should make students aware of the failures of waterfall
approaches and deep functional specialization, and introduce alternate execution
perspectives and approaches such as Schwaber & Sutherland (2012), Narayam (2015),
and Anderson (2010). The work of Don Reinertsen (2009) on optimizing the flow of
work within R&D organizations generally is receiving much attention in the digital and
Agile worlds. A good, balanced overview of Agile and project management is found in
Cobb (2015).
Systems analysis and architecture

Traditional systems analysis and architecture courses may risk reinforcing waterfall
thinking, as they too often assume a large “batch” of analysis is going to precede any
development. Coplien and Bjornvig (2010) and Bente (2012) examine the intersection of
Agile and architecture. Architecture and design recommendations can be found in the
Site Reliability Engineering professional literature at a more technical level (Allspaw &
Robbins 2010; Limoncelli et al. 2014; Beyer et al. 2016). Classes on architecture should
cover the critiques of former Netflix CTO Adrian Cockcroft (Bloomberg 2014; Cockcroft
2014)
Software testing and quality assurance

It is recommended that software testing and QA courses demonstrate awareness – and
ideally practical application – of Agile practices such as
 test-driven development (Whittaker et al. 2012; Freeman & Pryce 2010; Beck
2003),
 continuous integration (Duvall et al. 2007), and
 continuous delivery (Humble & Farley 2011; Kim et al. 2016).
Usability and User Interface Design

Usability and user experience design courses may be well-suited to expanded coverage of
product management in its broader aspects. Contextual material such as from Blank
(2013) and Cagan (2008) may be suitable, as well as readings from the design thinking
literature (Peck 2016; Lockwood 2010).

Operational topics
Operational topics are often overlooked in computing curricula, perhaps because IT and
digital operations have been seen as too specific to particular technologies, industries
and/or firms. However, as we have suggested in this report, there is a rich and growing
body of literature on digital operations, indicative of a growing and well-grounded
consensus on the topic. Teaching the full lifecycle of digital delivery is important in a
world requiring fast product feedback (Bass et al. 2016). This may require new course
development.
Digital labs and simulations

Platform simulations can be an effective teaching tool. Abstract explanations of complex
system interactions (e.g. narratives accompanied by architectural diagrams) come alive
when supported with direct, hands-on experience. Fortunately, modern computing
hardware and Cloud infrastructure, combined with powerful open source software
opens up educational possibilities unimaginable even ten years ago.
It is possible to create and operate complex digital pipelines entirely as virtualized
infrastructure. Iqbal (2016) provides an account of doing so in the information security
domain, showing principles that can be more generally applied. The software required is
powerful, reliable, and free for academic use. Virtualization makes simulation of complex
IT systems for educational purposes financially feasible. Hands-on student
experimentation can be encouraged, as systems can be easily spun up, experimented with,
and destroyed at no cost.
While we do not endorse any particular products at this time, some faculty have found
that HashiCorp Vagrant and Oracle VirtualBox are a useful combination. Once installed,
launching any number of fully configured virtual machines (based on open source
operating systems such as Ubuntu or CentOS) from online repositories is exceedingly
simple and fast.
It is important to understand the virtualization model in use here. Many are familiar with
uses of virtualization that start with the time-consuming build and configuration of a
guest operating system from installation media (e.g. an ISO file). In contrast, repository-
based images represent already-configured, known-good machines that can be readily
downloaded from well-known public sources. In essence, this approach trades bandwidth
for configuration time - given reasonable network speed, it is far quicker to download a
400-mb pre-built Ubuntu image (or even lighter-weight container) than it is to build one
from installation media.
The advantages of doing this, however, must be balanced against the risk that a given
image is "stale," with outdated software requiring so much updating that the advantages
of speed are lost. Or a given image is corrupted or compromised in some way.
Availability of the repository is also a concern.
Once downloaded, such machine images can interoperate with each other in arbitrary
topologies, exactly as a set of physical machines or blades in a rack in a data center. They
can have software installed and reinstalled, and can be quickly launched, configured, and
destroyed.

There are three primary ways in which instructors can provide laboratory simulations to
students:
 Simulation on instructor/student laptop
 Simulation hosted on campus computer resources
o Virtualization/private cloud
o Swarm of inexpensive physical devices (e.g. Arduino, Raspberry Pi)
 Simulation hosted on Cloud service provider(s)
Laptop simulation
In the laptop simulation, the student must install all of the base enabler software, e.g.:
 Virtualization engine (e.g. VirtualBox)
 Image repository manager (e.g. Vagrant)
 Configuration manager (e.g. Chef)
An SSH client and text editor are also helpful.
Installing the software and running the laptop simulation requires that the student has a
sufficiently powerful laptop. Late-model computers with at least 8 gigabytes of RAM and
500 gigabytes of disk space have been shown to be sufficient.
Such a configuration can run 5 to 15 virtual machines, depending on configuration.
Campus compute-based simulation

In this approach, the instructor or department manages one or more high power servers,
where the base enablers are then installed (not on the students' laptops). The capacity of a
typical on-campus server varies greatly, but a 48 GB 2014 server is known to support 32
virtual machines representing 4 pipelines, with simultaneous use by up to 24 students.
In this configuration, X-windows can solve some security issues related to hardening the
VMs.
If the campus is running true private Cloud, the characteristics of this approach will more
closely resemble the next option.
Cloud service provider simulation

In this approach, the virtual infrastructure is entirely hosted by one or more off-campus
Cloud providers. Such providers have effectively infinite capacity, for a price, and
therefore reference systems of great complexity can be created.
Cloud providers also provide a variety of platform services that must be designed and
built into the other two forms of simulation. Using such services, however, risks vendor
lock-in.
Various Cloud providers have educational programs offering free resources to instructors,
but these offers are limited and require re-application.

Security concerns
There are security concerns with any broad use of virtualization. The most significant
security concern is that virtual machines should not be bridged to open networks without
suitable hardening, which may not be trivial. A second concern is that images
downloaded from the Internet can be infected with malware, but this should be less of an
issue if well-established repositories are used.
Licensing concerns
The ease with which software can be downloaded and installed also raises some
classroom concerns. Some vendors make software available but still restrict its use via
licensing terms. Some of this restricted software may be available through widely used
package managers. Instructors and their institutions should review the licensing terms of
any software used in virtual classroom environments. (There are no known issues for
very broadly used platforms such as Ubuntu Linux, but more specialized packages
require more due diligence).
Summary of options
Laptop On-campus Cloud
Network Each laptop must have Campus server must Clients must have medium
requirements high bandwidth have high bandwidth bandwidth to Cloud
connection to Internet connection or be pre- provider
loaded with appropriate
images (note that this
topology can work for
remote or offline
locations); clients must
have medium bandwidth
to server
Security VMs on laptop should VMs on server must not VMs must not be bridged
not be bridged to be bridged to external out of Cloud VPN/VLAN
external network network
Writing and curating laboratories

Developing full infrastructure as code platforms suitable for instructional labs is a
significant development effort. Fortunately, lab platforms can support a variety of
learning objectives.
However, there are still questions regarding faculty effort required to build and sustain
realistic simulations involving complex, multi-product integrations. A central
clearinghouse for curating useful educational lab platforms would be beneficial to
educators.

Laboratory ideas
Virtualization and Automated Software Management Lab:

At the simplest, simulate a network environment with a test/development box and
production box by using Oracle VirtualBox software (available for free). Use Vagrant
software (free) to install two virtual machines, one as the development box and another as
the production box. Configure both boxes as needed with Vagrant software. Create shell
scripts such as backups, updates, and software installs and push them to boxes at the
same time or one at a time using Vagrant to demonstrate automated software
management with scripts. Create software code on the development box and push it to the
production box.
Containers Lab:
Using Docker or other containers (e.g. LXC), demonstrate installation, configuration,
management and destruction of containers. Manage containers from the host machine
(how to look up processes running on each container, what applications are running, etc.).
Other topics can include controlling container resources, running application containers,
monitoring containers and others.
OpenStack lab
This lab would provide students with the basics of running OpenStack. It would be a
useful prerequisite for software-defined networking.
Software-defined networking lab

This lab would extend OpenStack to include network function virtualization techniques.
Networks of arbitrary scale could be simulated for students, something that has not
typically been seen as feasible in mainstream educational environments.
Internet of Things (IoT) lab

This lab would use small physical devices (e.g. Raspberry Pi or Arduino) and/or virtual
resources to simulate complex Internet of Things infrastructures.
DevOps pipeline lab

This lab would simulate an end to end continuous delivery pipeline, including
 Test-driven development
 Continuous integration
 Build management
 Package management
 Continuous deployment
 Monitoring
It could serve as a basis for the following labs:

Microservices lab
This lab would use containers as a primary development/delivery mechanism,
implementing a microservices architecture.
Clustering and load-balancing lab

This lab would demonstrate basic principles of horizontal scaling and load balancing.
CAP principles lab

This lab would demonstrate principles of the CAP (Consistency, Availability, Partition-
Tolerance) theorem through simulating various data workloads.
Performance testing lab

This lab would introduce load drivers of various kinds to test the performance of the
virtual systems.
Auto-scaling lab
This lab would extend the clustering and performance testing labs to demonstrate the
principle of auto-scaling
Security lab
Security hardening is an important practical topic that can be taught through creating
virtual environments configured with various software packages and then running open
source vulnerability scanners
Chaos Engineering lab

This lab would extend the clustering and load balancing lab to include modules from the
Netflix Simian Army.
Vendor relationships
Additionally, vendor relationship strategies are especially important for this topic.
Vendors such as Amazon and Microsoft see early student exposure as a marketing
strategy, and therefore offer Cloud capacity at reduced or even no cost. Academic
institutions should take advantage of these offers but be aware of vendor lock-in.
DevOps tools
 Ansible allows users to manage systems via the lowest common denominator,
SSH so there is no special configuration or setup, just write some playbooks and
point them at your hosts.
 Apache Mesos is a tool for cluster management and utilization for distributed
applications in a DevOps world.
 Atlas provides visibility into infrastructure, including servers, containers, and
virtual machines, in addition to configuration management and service discovery.

 Chef is ideal for DevOps provisioning, orchestration and configuration
management; it supports auto rollback to previous versions, and provides real-
time visibility into metrics and messages.
 Collectd/l are tools that gather and store statistics about systems for measuring the
values of multiple system metrics –they can monitor different parameters in
parallel.
 Consul is a useful tool for service discovery and configuration in modern –it can
provide internal DNS names for services, act as a broker to help you sign and
register names, and enable you to access service names instead of specific
machines.
 Dataloop.IO has some nice features around dashboards and self-healing in a
hosted monitoring market.
 Docker is a tool for DevOps automation of application deployment inside
software containers. It is useful for nearly any software project and infrastructure
environment, and is especially helpful for managing dependencies and version
inconsistencies. Docker containers wrap up a piece of software in a single,
portable unit that contains everything it needs to run: code, runtime, system tools,
system libraries and so on.
 Drie.co is a PaaS solution aimed at being cloud vendor agnostic and it allows the
end user to interact via API calls to deploy solutions in a secure manor across
multiple cloud vendors.
 ELK (Elasticsearch, Logstash, Kibana) Stack is useful collecting logs from all
services, applications, networks, tools, servers, etc. into a single, centralized
location for processing and business intelligence analysis.
 FPM is a tool with the ability for generating packages for RedHat or Debian style
systems in order to ensure consistency and agility.
 GitHub tool has great forking and pull request features, and plugins that can
connect with Jenkins to facilitate integration and deployment.
 GoCD is a tool for pipeline management and visualization.
 Graphite and Grafana are two useful graph and dashboard builder for visualizing
time series metrics of data from on-premise systems, public cloud, private cloud
and IoT systems.
 GrayLog is useful for visualizing logs from multiple streams, for data analysis
and reporting about the health of the system.
 GuardRail is a configuration-monitoring tool, with continuous supervision of the
machine configuration state.
 Jenkins is a tool with a great ecosystem of plugins and add-ons –it can be used to
build code, create Docker containers, run tons of tests, and push to
staging/production.
 Kubernetes is a container orchestration tool, which can provide the management
and control for making the best use of micro services deployments, and in some
cases traditional deployments using Pods.

 Microsoft System Center Operations Manager (SCOM) is a legacy enterprise
infrastructure monitoring and management tool that delivers comprehensive
coverage for the Microsoft ecosystem of assets and devices installed on internal
data centers or in the cloud. SCOM is useful for monitoring network and storage
along with Unix-based systems. Since SCOM is a deep monitoring tool that
collects data from many different systems, it generates a large volume of data and
alerts that can overwhelm systems administrators and IT managers.
 Monit is a tool for assuring that any given process on a machine is up and running
–it can be used to restart a failed application such as an Apache process.
 New Relic is a tool for real-time in monitoring and managing web and mobile
applications within cloud, hybrid and on-premise infrastructure.
 PagerDuty is a tool to help monitor the stability of your platform and services.
 Puppet is best for DevOps automation and administration, and is a highly
prescriptive, process-driven platform that does not allow shortcuts.
 SaltStack is an event-driven, cloud infrastructure, automation tool for automating
tasks within the devops workflow. SaltStack supports systems management for
data automation, server provisioning, cloud building, and application
configuration.
 ServiceNow is a comprehensive service management tool, including service level
management, financial management, incident management, availability
management, change and release management and more.
 SolarWinds is a tool for managing and monitoring performance of a multivendor
distributed network environment; metrics include performance, capacity and
downtime.
 Splunk is a tool for finding and fixing issues in real time across the application
lifecycle; it allows developers to visualize data from production environments
without having to access production machines. Splunk helps users to embrace
devops processes, including continuous integration and deployment.
 Vistara is a comprehensive tool for monitoring heterogeneous multivendor assets
and devices installed on-premise or in the cloud. It covers a wide swath of
Microsoft and UNIX applications and systems, major enterprise vendors such as
Oracle and Symantec, and top storage and networking technologies. Vistara
records all user sessions, which provides an audit trail for tracing system changes
and other events.

Suggested resources
These resources are particularly useful as background for faculty and for in-class assignment:
Resource CA1 CA2 CA3 CA4 CA5
Dynamic Continuous Product Resource and Organization
infrastructure delivery management execution and culture
and operations management
Beck, et al.: The Agile x x
Manifesto
Kim: Phoenix project x x x x x
Morris: Infrastructure as Code x x
Beyer & Jones, Site x
Reliability Engineering
Allspaw: Web Operations x x
Limoncelli: Cloud Systems x x x
Admin
Chacon & Straub: Pro Git x
Patton: User Story Mapping x
Davis & Daniels: Effective x x x
DevOps
Bass: DevOps x x
Humble: Continuous Delivery x x
Ries: Lean Startup x
Gothelf: Lean UX x
Hussman (DevJam): Videos x
on Lean UX & Agile Product
Design
Anderson: Kanban x x
Cohn, Succeeding with Agile x x
Cobb, Project Manager’s x
Guide to Mastering Agile
Rother: Toyota Kata x
Gneezy, Meier, Rey-Biel: x x
When and Why Incentives
(Don’t) Work to Modify
Behavior
More in-depth resources, mostly useful as background (perhaps excerpted readings for in-class):
Resource CA1 CA2 CA3 CA4 CA5
Dekker: The Field Guide to Understanding x x
Human Error
Reinertsen: Principles of Product Development x x x
Flow
Mintzberg: Thinking in Fives x x
Fowler: Refactoring x
Hunt and Thomas: The Pragmatic Programmer x
Kahneman: Thinking Fast and Slow x x

Appendix A. Open issues
Issue Interested parties Notes

10/1/2016 Charles Betz, Aric Aune Product design (as opposed to discovery) was
identified as an open issue. However, the intent of
this document was to identify gaps, and there is
material in existing curricula on user experience
design and software architecture.
10/1/2016 Charles Betz (provenance of this The following was removed for lack of obvious
material unclear) citations.
Managing Lean product development in the

enterprise
 Identify concerns of multi-
product portfolios
 Identify potential and issues of
service re-use in the enterprise
product portfolio.
10/15/2016 Dave Lounsbury Internet of Things – treatment in CA1?

10/15/2016 Dave Lounsbury, Charles Betz Relationship of enterprise architecture to product
management.

Appendix B. References
This report is a response to industry influences. Peer-reviewed literature takes time to
catch up, so we have based much of our work on industry and professional sources. In
part our intention with the resources listed here is to curate a reliable baseline set for
digital curriculum development. Some of the material may appear tendentious, as Agile
often manifests as a disruptive reaction against older ways. Some references may be
coupled to particular technologies, some may use particular technologies as a basis for
understanding broader concepts, and some are of a general nature.
Abbott, Martin L.; Fisher, M.T., 2015. The Art of Scalability: Scalable Web Architecture, Processes, and
Organizations for the Modern Enterprise (2nd Edition), Old Tappan, NJ, NJ: Pearson Education,
Inc.
Acharya, S., 2014. Mastering Unit Testing Using Mockito and JUnit, Birmingham, UK, UK: Packt
Publishing, Ltd.
Adzic, G., 2012. Impact Mapping: Making a big impact with software products and projects, Gojko
Adzic.
Agile Alliance, 2001. Agile Manifesto and Principles. , (4/13/2011). Available at:
http://agilemanifesto.org/principles.html [Accessed January 1, 2016].
Agile Alliance, 2016. Subway Map to Agile Practices. Available at:
https://www.agilealliance.org/agile101/subway-map-to-agile-practices/ [Accessed August 11, 2016].
Allspaw, J., 2012. Blameless PostMortems and a Just Culture. Available at:
http://codeascraft.com/2012/05/22/blameless-postmortems/.
Allspaw, J. & Robbins, J., 2010. Web operations 1st ed., Beijing China ; Sebastopol, CA: O’Reilly.
Amazon Web Services, 2016a. Architecting for The Cloud: Best Practices. Available at:
https://aws.amazon.com/whitepapers/architecting-for-the-aws-cloud-best-practices/ [Accessed
September 10, 2016].
Amazon Web Services, 2016b. AWS Security Best Practices. Available at:
https://aws.amazon.com/whitepapers/aws-security-best-practices/ [Accessed September 10, 2016].
Ambler, S.W. & Lines, M., 2012. Disciplined Agile Delivery: A Practitioner’s Guide to Agile Software
Delivery in the Enterprise, Available at: http://www.ambysoft.com/books/dad.html.
Ambler, S.W. & Sadalage, P.J., 2006. Refactoring databases : evolutionary database design, Harlow,
U.K.: Addison-Wesley.
Anderson, D.J., 2010. Kanban: Successful Evolutionary Change for your Technology Business, Sequim,
WA: Blue Hole Press.
Andreessen, M., 2011. Why Software is Eating The World. The Wall Street Journal. Available at:
http://www.wsj.com/articles/SB10001424053111903480904576512250915629460 [Accessed
October 17, 2016].
Apke, L., 2015. Managers from Hell, The PMO is Dead, and Other Agile Stories, Amazon Digital
Services LLC.
Arbogast, T., Vodde, B. & Larman, C., 2012. Agile Contracts Primer. Available at:
http://www.agilecontracts.com/agile_contracts_primer.pdf.

Arnold, J., 2013. Tilt the playing field: discover, nurture, and speed up the delivery of value. Available at:
https://liber.io/v/liberioEpub_53aa4d19e3a7e.
Balter, B., 2015. The difference between 18F and USDS. Ben Balter blog. Available at:
http://ben.balter.com/2015/04/22/the-difference-between-18f-and-usds/ [Accessed April 21, 2016].
Bartram, A., 2015. Who Y Combinator Companies Want. Triplebyte: Building a better hiring process.
Available at: http://blog.triplebyte.com/who-y-combinator-companies-want#.nsoun8hrz [Accessed
October 21, 2016].
Bartsch, S., 2011. Practitioners’ Perspectives on Security in Agile Development. In ARES ’11
Proceedings of the 2011 Sixth International Conference on Availability, Reliability and Security.
IEEE CS. Available at:
https://pdfs.semanticscholar.org/6b55/d4f97760b051ef83a375d989f621bfcc39df.pdf [Accessed
October 20, 2016].
Bass, L. et al., 2016. Why You Should Be Teaching Deployment in Your Curriculum! In The 29th IEEE
International Conference on Software Engineering Education and Training. Dallas, Texas.
Available at: http://paris.utdallas.edu/cseet16/slides/02-CSEE&T-2016-Panel-I.pdf [Accessed
October 22, 2016].
Bass, L., Weber, I. & Zhu, L., 2015. DevOps: A Software Architect’s Perspective, New York: Addison
Wesley.
Beck, K., 1999. Embracing Change with Extreme Programming. IEEE Computer, 32(10), pp.70–77.
Beck, K., 2003. Test-Driven Development By Example, Available at:
http://books.google.com/books?hl=en&lr=&id=goE1TL0u0dUC&oi=fnd&pg=PR9&dq=Test-
Driven+Development+By+Example&ots=I9zvvKjuGi&sig=zSDvHh2aa5OrmQcVh7ppTTR5CKE.
Bente, S., Bombosch, U. & Langade, S., 2012. Collaborative Enterprise Architecture: Enriching EA with
Lean, Agile, and Enterprise 2.0 Practices, Waltham, MA: Morgan Kaufman - Elsevier.
Beyer, B. et al., 2016. Site Reliability Engineering: How Google Runs Production Systems, Sebastopol,
CA, CA: O’Reilly Media, Inc.
Binder, R.V., 1997. Can a manufacturing quality model work for software? IEEE Software, 14(5),
pp.101–102, 105. Available at:
http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=605937 [Accessed June 9, 2016].
Blank, S., 2013. The Four Steps to the Epiphany: Successful Strategies for Products That Win 2nd ed.,
Steve Blank.
Bloomberg, J., 2014. Agile Enterprise Architecture Finally Crosses the Chasm. Forbes. Available at:
http://www.forbes.com/sites/jasonbloomberg/2014/07/23/agile-enterprise-architecture-finally-
crosses-the-chasm/print/ [Accessed November 12, 2015].
Blumberg, M., 2013. Startup CEO: A Field Guide to Scaling Up Your Business, + Website, Wiley.
Boehm, B., 1988. A Spiral Model of Software Development and Enhancement. IEEE Computer, 21(5),
pp.61–72.
Bollinger C., T.B.. M., 1991. A Critical Look at Software Capability Evaluations. IEEE Software, 8(4),
pp.25–41.
Bond, J., 2015. The Enterprise Cloud: Best Practices for Transforming Legacy IT, Sebastopol, CA, CA:
O’Reilly Media, Inc.

Bonér, J. et al., 2014. The Reactive Manifesto. Available at: http://www.reactivemanifesto.org/ [Accessed
October 22, 2016].
Booch, G., 1996. Object Solutions: Managing the Object-Oriented Project, The Addison-Wesley
Publishing Company, Inc.
Bouthillier, L., 2005. Amazon.com’s Two-Pizza Team Rule. Available at:
http://www.learningapi.com/blog/archives/000079.html.
Brewer, E. & Fox, A., 1999. Harvest, Yield and Scalable Tolerant Systems. 7th Workshop Hot Topics in
Operating Systems (HotOS 99).
Brill, S., 2014. Obama’s Trauma Team. Time Magazine. Available at: http://blog.newrelic.com/wp-
content/uploads/80893.pdf [Accessed October 22, 2016].
Brown, W.J., 1998. AntiPatterns : refactoring software, architectures, and projects in crisis, New York:
Wiley. Available at: http://www.loc.gov/catdir/bios/wiley041/97039023.html.
Burgess, M., 2004a. Analytical network and system administration : managing human-computer
networks, Chichester, England: John Wiley & Sons Inc.
Burgess, M., 2004b. Principles of Network and System Administration 2nd ed., West Sussex, England,
England: John Wiley & Sons Ltd.
Burrows, M., 2015. Kanban from the Inside: Understand the Kanban Method, connect it to what you
already know, introduce it with impact, Sequim, Washington: Blue Hole Press.
Cagan, M., 2008. Inspired: How to Create Products Customers Love, SVPG Press. Available at:
http://www.amazon.com/Inspired-Create-Products-Customers-Love/dp/0981690408.
Cantor, M., 2011. Calculating and Improving ROI in Software and System Programs. Communications of
the ACM, 54(9), pp.121–130.
Chacon, S., 2011. Github Flow. Available at: http://scottchacon.com/2011/08/31/github-flow.html
[Accessed October 22, 2016].
Chacon, S., 2009. Pro Git, Berkeley, CA: Apress-Springer-Verlag.
Choi, J., 2014. The Science Behind Why Jeff Bezos’s Two-Pizza Team Rule Works. Available at:
http://blog.idonethis.com/two-pizza-team/ [Accessed October 22, 2016].
Cobb, C.G., 2015. The Project MANAGER’S GUIDE TO MASTERING AGILE: Principles and Practices
for an Adaptive Approach, Hoboken, New Jersey: John Wiley & Sons.
Cockburn, A., 2001. Writing effective use cases, Boston: Addison-Wesley.
Cockcroft, A., 2014. Fast Delivery DevOps Israel. Slideshare.net. Available at:
http://www.slideshare.net/adriancockcroft/fast-delivery-devops-israel [Accessed October 6, 2016].
Cohn, M., 2010. Succeeding with Agile: Software Development Using Scrum, Upper Saddle River, New
Jersey: Addison-Wesley.
Coplien, J. & Bjornvig, G., 2010. Lean Architecture for Agile Software Development, Chichester,
England: John Wiley & Sons Ltd.
Davis, J. & Daniels, K., 2016. Effective DevOps: Building a Culture of Collaboration, Tooling, and
Affinity at Scale, Sebastopol, CA, CA: O’Reilly Media, Inc.
Dekker, S., 2014. The Field Guide to Understanding “Human Error,” Burlington, VT, VT: Ashgate
Publishing Company.

Dekker, S., 2006. The Field Guide to Understanding Human Error, Available at:
http://www.amazon.com/Field-Guide-Understanding-Human-Error/dp/0754648265.
DeLuccia, J. et al., 2015. DevOps Audit Defense Toolkit,
Duvall, P.M., Matyas, S. & Glover, A., 2007. Continuous integration : improving software quality and
reducing risk, Upper Saddle River, NJ: Addison-Wesley. Available at:
http://www.loc.gov/catdir/toc/ecip0714/2007012001.html.
Farcic, V., 2016. The DevOps 2.0 Toolkit, LeanPub.
Forsgren, N. et al., 2016. 2016 State of DevOps report,
Forsgren, N., 2016. How Metrics Shape Your Culture. In Monitorama PDX 2016. Available at:
https://vimeo.com/173607646 [Accessed September 29, 2016].
Fowler, M. & Beck, K., 1999. Refactoring : improving the design of existing code, Reading, MA:
Addison-Wesley.
Freeman, S. & Pryce, N., 2010. Growing Object-Oriented Software, Guided by Tests, Upper Saddle
River, NJ, NJ: Pearson Education, Inc.
Furr, P. & Ahlstrom, N., 2013. Nail It then Scale It: The Entrepreneur’s Guide to Creating and Managing
Breakthrough Innovation, NISI Publishing.
Gilb, T., 1977. Software Metrics, Winthrop Publishers.
Gneezy, U., Meier, S. & Rey-Biel, P., 2011. When and Why Incentives (Don’t) Work to Modify
Behavior. Journal of Economic Perspectives—Volume, 25(4—Fall), pp.191–210.
Goldratt, E.M. & Cox, J., 2004. The goal : a process of ongoing improvement 3rd rev., Great Barrington,
MA: North River Press.
Goransson, P. & Black, C., 2014. Software Defined Networks: A Comprehensive Approach, Waltham,
MA, MA: Morgan Kaufman - Elsevier.
Graupp, P. & Wrona, R.J., 2011. Implementing TWI: Creating and Managing a Skills-Based Culture,
Boca Raton, FL: Productivity Press - Taylor & Francis Group.
Grove, A.S., 1995. High Output Management, New York, N.Y: Vintage - Penguin/Random House.
Gruver, G., Young, M. & Fulghum, P., 2013. A Practical Approach to Large-Scale Agile Development:
How HP Transformed Laserjet Futuresmart Firmware, Upper Saddle River, N.J.: Pearson
Education, Inc.
Hamilton, J., 2007. On Designing and Deploying Internet-Scale Services. In Proceedings of the 21st
Large Installation System Administration Conference (LISA ’07). Dallas, TX, TX. Available at:
https://www.usenix.org/legacy/event/lisa07/tech/full_papers/hamilton/hamilton_html/ [Accessed
October 22, 2016].
Harnish, V., 2014. Scaling Up: How a Few Companies Make It...and Why the Rest Don’t (Rockefeller
Habits 2.0) Gazelles,.,
Hashimoto, M., 2013. Vagrant: Up and Running, Sebastopol, CA, CA: O’Reilly Media.
Heller, M., 2016. GE’s Jim Fowler on the CIO role in the digital industrial economy. CIO Magazine
Online. Available at: http://www.cio.com/article/3048805/leadership-management/ges-jim-fowler-
on-the-cio-role-in-the-digital-industrial-economy.html [Accessed April 18, 2016].

Hernandez, J.G., 2012. Jeff Bezos’ Mandate: Amazon and Web Services. Psychology, Management and
Leadership. Available at: http://www.jesusgilhernandez.com/2012/10/18/jeff-bezos-mandate-
amazon-and-web-services/ [Accessed October 22, 2016].
Holzner, S., 2009. Ant: The Definitive Guide, Seb: O’Reilly Media.
Housman, M. & Minor, D., 2015. Toxic Workers, Available at: http://www.hbs.edu/faculty/Publication
Files/16-057_d45c0b4f-fa19-49de-8f1b-4b12fe054fea.pdf [Accessed June 28, 2016].
HP, 2015. Agile is the New Normal: Adopting Agile project management. Available at:
https://www.hpe.com/h20195/v2/getpdf.aspx/4AA5-7619ENW.pdf?ver=1. [Accessed September
30, 2016].
Humble, J. & Farley, D., 2011. Continuous delivery, Boston: Addison-Wesley.
Humble, J. & Molesky, J., 2011. Why Enterprises Must Adopt Devops to Enable Continuous Delivery.
Cutter IT Journal, 24(8).
Humphrey, W.S., 1989. Managing the software process, Reading, Mass.: Addison-Wesley.
Iqbal, S., 2016. Design and Emergence of a Pedagogical Online InfoSec Laboratory as an Ensemble
Artifact. Journal of Information Systems Education, 27(1), pp.17–35.
Izrailevsky, Y. & Tseitlin, A., 2011. The Netflix Simian Army. The Netflix Tech Blog. Available at:
http://techblog.netflix.com/2011/07/netflix-simian-army.html [Accessed May 4, 2016].
Jeff Gothelf & Josh Seiden, 2013. Lean UX: Applying Lean Principles to Improve User Experience,
Sebastopol, CA: O’Reilly Media, Inc.
Kaczanowski, T., 2013. Practical Unit Testing with JUnit and Mockito, Tomasz Kaczanowski. Available
at: https://www.amazon.com/Practical-Unit-Testing-JUnit-
Mockito/dp/8393489393/ref=asap_bc?ie=UTF8.
Kahn, K.B. & Product Development & Management Association., 2005. The PDMA handbook of new
product development 2nd ed., Hoboken, N.J.: Wiley. Available at:
http://www.loc.gov/catdir/description/wiley042/2004002261.html.
Kennaley, M., 2010. Sdlc 3.0: Beyond a Tacit Understanding of Agile: Towards the Next Generation of
Software Engineering, Fourth Medium Consulting.
Kim, G. et al., 2016. The DevOps Handbook, Portland, OR, OR: IT Revolution Press.
Kim, G., Behr, K. & Spafford, G., 2013. The Phoenix Project: A Novel About IT, DevOps, and Helping
Your Business Win, IT Revolution Press.
Kniberg, H., 2016. Spotify Rhythm – how we get aligned. Available at:
http://blog.crisp.se/2016/06/08/henrikkniberg/spotify-rhythm [Accessed August 31, 2016].
Kniberg, H. & Ivarsson, A., 2012. Scaling Agile @ Spotify with Tribes, Squads, Chapters & Guilds.
Available at: https://dl.dropboxusercontent.com/u/1018963/Articles/SpotifyScaling.pdf [Accessed
October 20, 2016].
Kohavi, R., Crook, T. & Longbotham, R., 2009. Online Experimentation At Microsoft. Available at:
http://www.exp-platform.com/Documents/ExPThinkWeek2009Public.pdf [Accessed August 31,
2016].
Koskela Gregory, L.H., 2002. The underlying theory of project management is obsolete. Available at:
http://www.researchgate.net/publication/3229647_The_Underlying_Theory_of_Project_Manageme
nt_Is_Obsolete [Accessed April 17, 2016].

Labs, G.K.& J.H.& D.N.F.& P., 2014. 2014 Puppet Labs DevOps Report. Available at:
http://puppetlabs.com/2014-devops-report.
Larman, C. & Basili, V.R., 2003. Iterative and incremental development: A brief history. Computer,
36(6), pp.47–56.
Larman, C. & Vodde, B., 2009. Scaling lean & agile development : thinking and organizational tools for
large-scale Scrum, Upper Saddle River, NJ: Addison-Wesley.
Laudon, K.C. & Laudon, J.P., 2015. Management Information Systems : Managing the Digital Firm - 9th
edition,
Lee, T., 2016. Here’s Obama’s plan to prevent future IT disasters like the Healthcare.gov rollout. Vox
Technology. Available at: http://www.vox.com/2016/4/20/11439754/agile-programming-it-reform
[Accessed April 20, 2016].
Leffingwell, D., 2011. Agile software requirements : lean requirements practices for teams, programs,
and the enterprise, Upper Saddle River, NJ: Addison-Wesley.
Leffingwell, D. et al., 2014. Scaled Agile Framework. , (10 October 2015). Available at:
http://scaledagileframework.com/.
Lencioni, P., 2002. The Five Dysfunctions of a Team: A Leadership Fable, San Francisco, CA: Jossey-
Bass (Wiley).
Lenfle, S. & Loch, C., 2010. Lost Roots: How Project Management Came to Emphasize Control Over
Flexibility & Novelty. California Management Review, 53(1), pp.32–56. Available at:
http://www.jstor.org/stable/10.1525/cmr.2010.53.1.32.
Limoncelli, T., 2012. SRE at Google: Thousands Of DevOps Since 2004 At Google. Available at:
http://www.youtube.com/watch?v=iIuTnhdTzK0.
Limoncelli, T.A., Chalup, S.R. & Hogan, C.J., 2014. The Practice of Cloud System Administration:
Designing and Operating Large Distributed Systems, Vol. 2, Pearson Education. Available at:
http://www.amazon.com/Practice-Cloud-System-Administration-Distributed/dp/032194318X/.
Lockwood, T., 2010. Design Thinking: Integrating Innovation, Customer Experience, and Brand Value,
New York, N.Y.: Allworth Press - Allworth Communications.
Matthias, K., 2015. Docker: Up and Running, Sebastopol, CA, CA: O’Reilly Media, Inc.
Memon, T.F., 2014. Project vs. Product. Available at:
https://www.thoughtworks.com/insights/blog/project-vs-product.
Millard, C. ed., 2013. Cloud Computing Law, Oxford, UK, UK: Oxford University Press.
Morris, K., 2016. Infrastructure as Code: Managing Servers in the Cloud, Sebastopol, CA, CA: O’Reilly
Media, Inc.
Narayam, S., 2015. Agile IT organization design: for digital transformation and continuous delivery,
Pearson Education Inc. .
Newman, S., 2015. Building Microservices, Sebastopol, CA: O’Reilly Media, Inc.
NIST, 2011. NIST SP 800-145, The NIST Definition of Cloud Computing, Available at:
http://dx.doi.org/10.6028/NIST.SP.800-145 [Accessed April 5, 2016].
Nygard, M.T., 2007. Release it! : design and deploy production-ready software, Raleigh, N.C.: Pragmatic
Bookshelf. Available at: http://www.loc.gov/catdir/enhancements/fy0728/2007277208-d.html.

Ogunnaike, B.A. & Ray, W.H., 1994. Process Dynamics, Modeling, and Control, Oxford: Oxford
University Press.
Opelt, A. et al., 2013. Agile contracts : creating and managing successful projects with Scrum,
Open Group IT4IT Forum Agile Workgroup, 2016. IT4ITTM Agile Scenario: Using the Reference
Architecture, Available at: https://www2.opengroup.org/ogsys/catalog/W162.
Osterwalder, A. & Pigneur, Y., 2010. Business Model Generation. Wiley, p.280. Available at:
http://www.businessmodelgeneration.com/canvas.
Patton, J., 2014. User Story Mapping. Discover the Whole Story, Build the Right Product First edit.,
Available at: http://shop.oreilly.com/product/0636920033851.do.
Peck, C., 2016. U expert tells how “design thinking” can solve society’s big problems. Minnesota Star
Tribune.
Pentland, S., Pentland, A.S. & Performance, H., 2012. The New Science of Building Great Teams
Analytics for Success The New Science of Building Great Teams : Analytics for Success. Harvard
Business Review, 90(April), pp.60–69.
Poppendieck, M. & Poppendieck, T.D., 2007. Implementing lean software development : from concept to
cash, London: Addison-Wesley. Available at:
http://www.loc.gov/catdir/toc/ecip0615/2006019698.html.
Poppendieck, M. & Poppendieck, T.D., 2003. Lean Software Development: An Agile Toolkit A. Cockburn
& J. Highsmith, eds., Boston: Addison Wesley.
Portnoy, M., 2016. Virtualization Essentials, Indianapolis, Indiana, Indiana: John Wiley & Sons, Inc.
Puppet Labs, 2015. 2015 State of DevOps Report,
Reinertsen, D.G., 1997. Managing the design factory: a product developer’s toolkit, New York ; London:
Free Press.
Reinertsen, D.G., 2009. The principles of product development flow: second generation lean product
development, Redondo Beach, Calif.: Celeritas.
Ries, E., 2011. The lean startup : how today’s entrepreneurs use continuous innovation to create
radically successful businesses 1st ed., New York: Crown Business.
Rigby, D.K., Sutherland, J. & Takeuchi, H., 2016. Embracing Agile. Harvard Business Review, (May).
Available at: https://hbr.org/2016/05/embracing-agile.
Rother, M. & Shook, J., 2003. Learning to See: Value Stream Mapping to Add Value and Eliminate
MUDA [Spiral-bound]. Lean Enterprise Institute, p.102.
Rozovsky, J., 2015. The five keys to a successful Google team. re:Work. Available at:
https://rework.withgoogle.com/blog/five-keys-to-a-successful-google-team/.
Rubin, K.S., 2012. Essential Scrum : a practical guide to the most popular agile process, Upper Saddle
River, NJ: Addison-Wesley.
Rummler, G.A. & Brache, A.P., 1995. Improving performance: how to manage the white space on the
organization chart 2nd ed., San Francisco, CA: Jossey-Bass. Available at:
http://www.loc.gov/catdir/toc/wiley041/94048105.html.
Ruparelia, N.B., 2016. Cloud Computing, Cambridge, Mass., Mass.: The MIT Press.
Schwaber, K., 2002. Agile Software Development with Scrum, Upper Saddle River, N.J.: Prentice Hall.

Schwaber, K., 2007. The Enterprise and Scrum, Redmond, Wash: Microsoft Press.
Schwaber, K. & Sutherland, J.V., 2012. Software in 30 days : how agile managers beat the odds, delight
their customers, and leave competitors in the dust, Hoboken, N.J.: John Wiley & Sons, Inc.
Available at: http://catalogimages.wiley.com/images/db/jimages/9781118206669.jpg.
Schwartz, M., 2014. How DevOps Can Fix Federal Government IT. DevOps Enterprise.
Sean Landis, 2011. Agile Hiring, Artima, Inc.
Securosis, 2014. Secure Agile Development, Available at:
https://securosis.com/assets/library/reports/SecureAgileDevelopment_Nov2014_FINAL.pdf
[Accessed September 10, 2016].
Shackelford, R. et al., 2005. Computing Curricula 2005 The Overview Report, Available at:
http://www.acm.org/education/education/curric_vols/CC2005-March06Final.pdf [Accessed May 28,
2016].
Shortland, A. & Lei, M., 2012. Using Rundeck and Chef to build DevOps Toolchains. Available at:
http://dev2ops.org/2012/05/using-rundeck-and-chef-to-build-devops-toolchains-at-chefcon/
[Accessed October 20, 2016].
Singer, J., 2004. Meta Data Repository Redux. DM Review.
Smart, J.F., 2011. Jenkins: The Definitive Guide, Sebastopol, CA, CA: O’Reilly Media.
Smith, P.G. & Reinertsen, D.G., 1998. Developing products in half the time : new rules, new tools [New
ed. ]., New York ; London: Van Nostrand Reinhold.
Snowden, D.J. & Boone, M.E., 2007. A Leader’s Framework for Decision Making. Harvard Business
Review.
Sonatype, 2009. Maven: The Definitive Guide, Sebastopol, CA, CA: O’Reilly Media.
Spinellis, D., 2015. Extending Our Field’s Reach. IEEE Software, pp.4–6. Available at:
https://www.computer.org/csdl/mags/so/2015/06/mso2015060004.pdf [Accessed October 20, 2016].
Sussna, J., 2015. Designing Delivery: Rethinking IT in the Digital Service Economy, O’Relly
Publications.
Takeuchi, H. & Nonaka, I., 1986. The new new product development game. Journal of Product
Innovation Management, 3(3), pp.205–206. Available at: http://doi.wiley.com/10.1016/0737-
6782(86)90053-6.
Taylor, M. & Vargo, S., 2015. Learning Chef: A Guide to Configuration Management and Automation,
Sebastopol, CA, CA: O’Reilly Media.
The Open Group, 2015. The Open Group Architectural Framework (TOGAF), Version 9, Open Group,
The. Available at: http://www.opengroup.org/togaf/.
Topi, H. et al., 2016. Revising the MSIS Curriculum: Specifying Graduate Competencies (Second Public
Deliverable of the ACM/AIS MSIS 2016 Task Force), Available at:
http://cis.bentley.edu/htopi/MSIS2016_Draft_3-21-2016.pdf.
Vergun, D., 2015. Toxic leaders decrease Soldiers’ effectiveness, experts say. www.army.mil. Available
at:
https://www.army.mil/article/157327/Toxic_leaders_decrease_Soldiers__effectiveness__experts_sa
y [Accessed July 13, 2016].

Westerman, G. et al., 2011. Digital Transformation: A Road-Map for Billion-Dollar Organizations. MIT
Center for Digital Business and Capgemini Consulting, pp.1–68.
Westerman, G. et al., 2012. The Digital Advantage: How Digital Leaders Outperform their Peers in Every
Industry. MIT Sloan Management Review, pp.1–24. Available at:
http://www.capgemini.com/resource-file-
access/resource/pdf/The_Digital_Advantage__How_Digital_Leaders_Outperform_their_Peers_in_E
very_Industry.pdf [Accessed October 22, 2016].
Wheeler, D., 2000. Understanding Variation: The Key to Managing Chaos, SPC PRESS.
Whittaker, J.A., Arbon, J. & Carollo, J., 2012. How Google tests software, Upper Saddle River, NJ:
Addison-Wesley.
Wiggins, A., 2012. The Twelve-Factor App. Available at: http://12factor.net/ [Accessed August 11,
2016].
Wikipedia, 2016. DevOps. Available at: https://en.wikipedia.org/wiki/DevOps [Accessed May 18, 2016].
Williams, C., 2016. How one developer just broke Node, Babel and thousands of projects in 11 lines of
JavaScript • The Register. The Register (UK). Available at:
http://www.theregister.co.uk/2016/03/23/npm_left_pad_chaos/ [Accessed October 22, 2016].
Williams, D.P. & Holub, E., 2016. How to Build a DevOps Release Team, Available at:
https://www.gartner.com/doc/reprints?id=1-2YUEQCD&ct=160217&st=sb.


Renew The IT Curriculum

Uploaded by

Copyright:

Available Formats

You might also like

Renew The IT Curriculum

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Renew The IT Curriculum

Uploaded by

Copyright:

Available Formats

Renewing the IT Curriculum:

Responding to Agile, DevOps, and Digital Transformation

This report is an initiative of Advance IT Minnesota,

Renewing the IT Curriculum Page |i

Renewing the IT Curriculum Page |i

Every company needs IT to get things done.

Renewing the IT Curriculum P a g e | ii

Renewing the IT Curriculum P a g e | iii

Renewing the IT Curriculum P a g e | iv

Document control and citation

Renewing the IT Curriculum Page |v

Renewing the IT Curriculum P a g e | vi

Renewing the IT Curriculum Page |1

Renewing the IT Curriculum Page |2

Renewing the IT Curriculum Page |3

Renewing the IT Curriculum Page |4

Goals of this report

Renewing the IT Curriculum Page |5

Renewing the IT Curriculum Page |6

Renewing the IT Curriculum Page |7

Renewing the IT Curriculum Page |8

Renewing the IT Curriculum Page |9

Renewing the IT Curriculum P a g e | 10

Defining “IT,” “Agile,” “DevOps,” and “digital”

Information Technology (IT)

Renewing the IT Curriculum P a g e | 11

Agile and the Agile ecosystem

Renewing the IT Curriculum P a g e | 12

Renewing the IT Curriculum P a g e | 13

Renewing the IT Curriculum P a g e | 14

Approach to learning objectives

Renewing the IT Curriculum P a g e | 15

Renewing the IT Curriculum P a g e | 16

Renewing the IT Curriculum P a g e | 17

Renewing the IT Curriculum P a g e | 18

Software Defined Infrastructure

Version Control Basics

Renewing the IT Curriculum P a g e | 19

Automation & configuration

Renewing the IT Curriculum P a g e | 20

Availability and risk

Automation and operations

Designing for scale and failure

Renewing the IT Curriculum P a g e | 21

Causal analysis, mitigation, and remediation

Security and privacy

Renewing the IT Curriculum P a g e | 22

Agile and related software development practices

Core Agile practices

Renewing the IT Curriculum P a g e | 23

Continuous Delivery pipelines

Understanding the end to end digital pipeline

Managing the development environment

Renewing the IT Curriculum P a g e | 24

Code analysis and peer review

Renewing the IT Curriculum P a g e | 25

Deployment and integration testing