Building a NetDevOps Pipeline – Part I

Marcel Neidinger
Software Solutions Engineer, EMEAR Systems Engineering
30th of November, 2021
Where are we? And where are we going?
Intro to
What is NetDevOps?
NetDevOps

Model-Driven Controler-based
SSH. automated. How do we change?
Programmability automation

Version Control What do we push?

Virtual Lab pyATS for

How do we test?
Setup with CML testing

Building a full
advanced pyATS NetDevOps Pipeline How do we build a pipeline?

© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 2
 • A DevOps Recap
• From DevOps to NetDevOps
This week • An overview of our tools
• Designing our NetDevOps pipeline

• Implementing our NetDevOps Pipeline

Next week • Pushing Configuration using our new Pipeline

© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 3
 • A DevOps Recap
• From DevOps to NetDevOps
This week • An overview of our tools
• Designing our NetDevOps pipeline

• Implementing our NetDevOps Pipeline

Next week • Pushing Configuration using our new Pipeline

© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 4
A DevOps Recap

© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 5
 “Progress is impossible
without change […]”

George Bernard Shaw

American Playwright
© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 6
 “Move fast and break things.”

Mark Zuckerberg
Founder and CEO
Facebook Inc.
© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 7
 If you like your job
Don’t break
things.
© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 8
Culture of Fear Changes
happen rarely
All changes
Change seen are big and
as failure complex

Problem
The team isn’t
occurs during
practiced
change
Change is (seen as)
high risk
© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 9
Are we the

first?
© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 10
Software is the

same
© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 11
“A Software is a set of source code files that
together describe behaviour.

Changing a software means adding changes

to mission critical components who's
outage or failure can cost millions”
© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 12
What problem does DevOps solve?

© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 13
The teams

🧑‍💼🧑‍💻 🧑‍🔬 👷
Business Development Testing Operations

© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 14
Plan

🧑‍💼🧑‍💻 🧑‍🔬 👷
Business Development Testing Operations

© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 15
Development

🧑‍💼🧑‍💻 🧑‍🔬 👷
Business Development Testing Operations

© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 16
Build

🧑‍💼🧑‍💻 🧑‍🔬 👷
Business Development Testing Operations

© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 17
Test

🧑‍💼🧑‍💻 🧑‍🔬 👷
Business Development Testing Operations

© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 18
Release

🧑‍💼🧑‍💻 🧑‍🔬 👷
Business Development Testing Operations

© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 19
Deploy

🧑‍💼🧑‍💻 🧑‍🔬 👷
Business Development Testing Operations

© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 20
Operate

🧑‍💼🧑‍💻 🧑‍🔬 👷
Business Development Testing Operations

© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 21
Monitor

🧑‍💼🧑‍💻 🧑‍🔬 👷
Business Development Testing Operations

© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 22
Let’s put all these connections together

🧑‍💼🧑‍💻 🧑‍🔬 👷
Business Development Testing Operations

© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 23
 Different team culture Different workflows

Different ‘language’ Different deliverables

🧑‍💼🧑‍💻 🧑‍🔬 👷
Business Development Testing Operations

Different tools
Different priorities
Different metrics
Different targets
© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 24
50.000.000
deploys across all services *

© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential *That includes all development, testing and production systems25
Source:
The Story of Apollo - Amazon’s Deployment Engine, Werner Vogels (amazon CTO), https://www.allthingsdistributed.com/2014/11/apollo-amazon-deployment-engine.html
 changes
per second
© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 26
Source:
The Story of Apollo - Amazon’s Deployment Engine, Werner Vogels (amazon CTO), https://www.allthingsdistributed.com/2014/11/apollo-amazon-deployment-engine.html
 Continous Integration Continous Delivery
© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 27
A CI/CD Pipeline
💻 🏗️ 📦
Commit Changes Changes are Build Deliver changes
Changes to the source A new version of the A new version of the
code are commited to software is build. software is delivered.
the VCS.

🚧 📋 🚀
Trigger a build Run tests Deploy to infrastructure
Change is detected The test suite is run Roll out the new version
and a new build is agains the newly build to all servers. Either fully or
triggered. software version. in incremental steps.

© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 28
 Continous Integration
A CI/CD Pipeline
💻 🏗️ 📦
Commit Changes Changes are Build Deliver changes
Changes to the source A new version of the A new version of the
code are commited to software is build. software is delivered.
the VCS.

🚧 📋 🚀
Trigger a build Run tests Deploy to infrastructure
Change is detected The test suite is run Roll out the new version
and a new build is agains the newly build to all servers. Either fully or
triggered. software version. in incremental steps.

© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 29
 Continous Integration
A CI/CD Pipeline Continous Delivery

💻 🏗️ 📦
Commit Changes Changes are Build Deliver changes
Changes to the source A new version of the A new version of the
code are commited to software is build. software is delivered.
the VCS.

🚧 📋 🚀
Trigger a build Run tests Deploy to infrastructure
Change is detected The test suite is run Roll out the new version
and a new build is agains the newly build to all servers. Either fully or
triggered. software version. in incremental steps.

© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 30
 Mindset Change
changes happens rarely changes happens often

small
rollback big change fail fast Change

Culture Culture
Of Of
FEAR CHANGE

not well low risk well

high risk practiced
practiced

© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 31
“DevOps is a set of practices that combines
software development (Dev) and IT
operations (Ops).

It aims to shorten systems development life

cycle and provide continuous delivery with
high software quality.
© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 32

Source: wikipedia.org/wiki/DevOps
From DevOps to NetDevOps

© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 33
“A Software is a set of source code files that
together describe behaviour.

Changing a software means adding changes

to mission critical components who's
outage or failure can cost millions”
© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 34
 Network Configuration
“A Software is a set of source code files that
together describe behaviour.
Network
Changing a software means adding changes
to mission critical components who's
outage or failure can cost millions”
© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 35
 “NetDevOps brings the
culture, technical methods,
strategies, and best practices
of DevOps to Networking”
Hank Preston
Network Programmability Evangelist
Cisco DevNet
© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 36
 Harder to push updates

Harder to “spin up” version for testing

Infrastructure
is not equal to
Code
Harder to define changes
Harder to test without device access

© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 37
 Infrastructure
as
Code
© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 38
What is NetDevOps then?
Deploy Test
Continuous Delivery Automated Validation

IN F R A S T R U C T U R E
AS CODE

Design Rollout
Collaborative Continuous Operations
Development
© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 39
Adopting the CI/CD Pipeline for NetDevOps
💻 🏗️ 🚀
Commit Changes Changes are Build Deploy to infrastructure
Changes to our A new version of Roll out the new version
configuration template our device-specific of our configuration to
is commited. configuration is build. all devices.

🚧 📋
Trigger a build Run tests
Change is detected The test suit is run
and a new build is against a simulated or lab
triggered. version of our network.

© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 40
An Overview of our Tools

© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 41
NetDevOps Tools

Design Configuration
Collaborate Management

"
Stateful
Lab/Simulation
Test &
CML Environment
Validation

© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 42
 DevOps Tools

© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 43
 💻 Commit Changes

© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 44
 💻 Build Changes

© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 45
 Trigger a build
🚧
© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 46
 Changes are build
🏗️
© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 47
 CML

Run Tests
📋
© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 48
 Deploy to Infrastructure
🚀
© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 49
Designing our NetDevOps
pipeline

© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 50
 💻 🚧
© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
🏗️ 📋 🚀 51
Implement a change to our
configuration template
• Write a change to our jinja2
template
• Test generated config locally

💻 🚧
© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
🏗️ 📋 🚀 52
Implement a change to our
configuration template
• Push our changed config template
Push this change to our to git
VCS and trigger build • Use gitlab CI/CD runner to trigger
build after commit
• Pipeline is specified in
.gitlab-ci.yml

💻 🚧
© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
🏗️ 📋 🚀 53
Implement a change to our
configuration template
• Pipeline compiles templates
Push this change to our • Device-specifc context is retrieved
VCS from NetBox
• Config is rendered using context
Pull information from from NetBox
NetBox and build config

💻 🚧
© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
🏗️ 📋 🚀 54
Implement a change to our
configuration template
• Configure new testing lab in CML
Push this change to our (using API)
VCS • Configuration changes are applied
to virtualized lab
Pull information from
NetBox and build config

Run Config changes against

CML

💻 🚧
© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
🏗️ 📋 🚀 55
Implement a change to our
configuration template
• Retrieve testbed from CML API
Push this change to our • pyATS tests are run against the
VCS CML-based simulated
environment using testbed
Pull information from
NetBox and build config

Run Config changes against

CML

Run pyATS tests against

CML

💻 🚧
© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
🏗️ 📋 🚀 56
Implement a change to our
configuration template
• Generate production testbed from
Push this change to our NetBox
VCS • Run pyATS changes against
production testbed
Pull information from
NetBox and build config

Run Config changes against

CML

Run pyATS tests against Deploy changes to

CML production

💻 🚧
© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
🏗️ 📋 🚀 57
 • A DevOps Recap
• From DevOps to NetDevOps
This week • An overview of our tools
• Designing a NetDevOps pipeline

• Implementing our NetDevOps Pipeline

Next week • Pushing Configuration using our new Pipeline

© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 58