SDN WAN Orchestration

Principles and Solutions

Josh Peters, CVG, joshpete@cisco.com
Chris Metz, CTAO, chmetz@cisco.com
Rached Blili, CVG, rachedb@cisco.com

CiscoLive 2015 – San Diego

TECMPL-3200
Contributors
• Santiago Alvarez
• Alan Gous
• Derek Tay
• Joe Malcolm
• Arash Afrakhteh
• Eleni Palkopoulou
• James Lang
• John Lehane
• Marcelo Silva
• Jan Medved
• Juraj Sebin
• Max Williams
Agenda – Part 1
• Introduction & Scope
• Architecture & Components
• WAN Orchestration Solutions #1
• Capacity Planning
• Failure Impact Analysis
• Multi-Layer Optimization
• Trending Analysis, Health Analysis
• Coordinated Maintenance
• Summary & Part 2 Setup
• Break
Agenda – Part 2
• Protocol & API Primer
• WAN Orchestration Solutions #2
• Bandwidth on Demand
• Bandwidth Scheduling
• Minimize Congestion in a Segment Routed Network
• Tunnel Split/Merge
• Tunnel Balancer
• Intelligent Workload Placement
• Reactive Applications
• Practical Implementation Steps
• Summary & References
Introduction and Scope
Let’s Start with some definitions - WAN
• Wikipedia says:
• “.. a network that covers a broad area (i.e.,
any telecommunications network that links
across metropolitan, regional, or international
boundaries) using leased telecommunication
lines ..” WAN
• Network of heterogeneous packet
forwarding vehicles (e.g. routers) IP/MPLS
managed by an operator supporting
multiple services
• Service Provider or Enterprise Optical
• Single or multi-layer
• IP and/or label-based forwarding 6
What is WAN Orchestration?
• Wikipedia says for Orchestration:
• “.. automated arrangement, coordination, and management of complex
computer systems, middleware, and services..”
• If applied to a WAN what are we orchestrating? Examples are:

Image of WAN +
ER2 CR4
cloud services
CR5 CR6

ER3

Device Configuration Path/traffic Placement Cloud + WAN

 Reference

Trends Impacting the WAN (1)

Cloud Growth

Cloud Resources Connected via WAN

Reference: http://www.cisco.com/c/en/us/solutions/service-provider/global-cloud-index-gci/index.html
Trends Impacting the WAN (2) – Reference
Video. Enough Said

Reference: http://variety.com/2015/digital/news/ott-map-video-ecosystem-1201480930/
Software Defined Networks (SDN)
• Opennetworking.Org says: Apps
• “.. control and data planes are decoupled,
network intelligence centralized … network
infrastructure abstracted from the
NB API
applications…”
• Practical SDN attributes: Centralized Platform with
Global View
• Software platform with (abstracted) network
view
• Northbound APIs to/from applications
• Southbound methods to collect and program IP/MPLS
information from/to network
• Applicable to any network domain
Optical
A More Practical Definition of SDN

• Ivan Pepelnjak, ipSpace.net, “Software Defined Networks – Four Years Later”

https://ripe70.ripe.net/archives/#tue
 Reference

“SDN-ish” Action to Date by Place in the Network

Datacenter/
Cloud,
Campus,
Mobile, Cable

WAN

SDN present in many domains

Focus in SDN shifting to WAN (1)
• Critical bottleneck “underlay” component in the Cloud Service &
Networking Space
• IETF standardization work – stateful PCE, YANG Models, etc.
• Deployment of network technology conducive to SDN control
• Segment Routing, IP/Optical Networks

• Vendor Investment
• Hybrid Cloud/WAN Performance Engineering
• OpenDaylight initiative includes SDN WAN protocol support
Focus in SDN shifting to WAN (2)

(image)
Run Network
Hotter

Run the Network Hotter Reservations Togetherness [cloud, WAN]

Apps
APIs
Platform
Development Platform
Task Automation

Visibility
 Reference
ACG Research Business Case Study
SDN in the WAN

• 32% ROI or a bandwidth calendaring service

• 52% five-year CAGR for a bandwidth on
demand service
• 27% TCO savings for a TE tunnel balancing
solution
• 38% TCO savings for a TE capacity planning
solution

http://acgcc.com/business-case-for-cisco-sdn-for-the-wan-2/
Traffic Engineering Principles (1)
• Top level goal is to reduce the relative cost of network infrastructure
• Sweat Assets, maximize returns …
• Achieved by a variety of methods. Examples:
• Load balancing / Load splitting, Service differentiation, Congestion mitigation
• Capacity Planning

• Technologies applied include:

• IGP Metric Tuning
• QoS
• MPLS and Segment Routing TE
• Network Modeling/Simulation
• Over-Engineer
Sometimes TE methods can be less than precise …
Traffic Engineering Principles (2)
Tradeoffs

• WAN – complex architectural and operational complexity

• Lack of predictability on the network

• Leads to over-engineering
• Leads to big headaches when things break

• Planning cycles are (too) long

• Service velocity and TTM is longer
Why TE?
• WAN b/w is critical
• WAN resources are expensive
• Impact of failure is large
• Making physical changes is difficult
• Degree of skill needed is high

The ultimate goal is cost savings!

Achieved thru optimization and automation
Software Platforms for SDN WAN Orchestration

• Demand Admission
• Load Placement/Balancing
• Visualization • Bandwidth Calendaring
• Capacity Planning • Automated TE Engineering
• Failure Analysis • Multicast Tree Management
• Traffic Trending • Multi-layer Optimization
• Analytics • Re-optimization/Rearrangment
• Cost Modeling

Offline SDN WAN Orchestration Continuum Online

Objectives for Today

I’m lost. Show me!

Problem? We will show the architecture Running code
Here is a solution. and solution examples and demonstrations.
workflow.

What do I need? Need more details.

List of solution Ingredients Answer all of your Some background on
questions. Protocols and API
standards
SDN WAN Orchestration Platform GPS
Apps

Northbound APIs

Optimization &
Analytics Calendaring
Prediction

Current Network New

Model Modeler Model

Network
Collector Deployer
Interface
SNMP CLI Netflow BGP NMS/EMS Netconf/YANG PCEP
 Reference
SDN WAN Orchestration Platform GPS
Applications Apps

Northbound APIs

Optimization &
Analytics Calendaring
Prediction

Current Network New

SDN WAN Orch Platform Model Modeler Model

Network
Collector Deployer
Interface
SNMP CLI Netflow BGP NMS/EMS Netconf/YANG PCEP

Network(s)
SDN WAN Orchestration Platform GPS Solution Tagging

Apps
Example: Solution
involving SDN WAN
Northbound APIs Analytics will be tagged
with the color-coded
primary function blocks
Analytics

Current Network
Model Modeler

Network
Collector
Interface
SNMP CLI Netflow BGP NMS/EMS Netconf/YANG PCEP

App/API Network
 Scope
SDN WAN Orchestration Positioning End-to-End
Examine applications and solutions ** WAN X-Domain Orch Apps DC-WAN X-Domain Orch

for SDN WAN Orchestration

Other NB API
DC/Cloud
Domain
SDN
SDN Orchestration

Ent/Campus/ Collector Deployer DC/Cloud

Access/etc. Providers

Apps IPv4/IPv6/MPLS/SR

Northbound APIs
Optical
Optimization &
Analytics Calendaring
Prediction

Current Network New

Model Modeler Model

Network
Collector Deployer
Interface
SNMP CLI Netflow BGP NMS/EMS Netconf/YANG PCEP

SDN WAN Orchestration

Platform GPS to help explain
functions, flows and solutions.
Architecture and
Components
SDN WAN Orchestration Platform GPS
Applications Apps

Northbound APIs

Optimization &
Analytics Calendaring
Prediction

Current Network New

SDN WAN Orch Platform Model Modeler Model

Network
Collector Deployer
Interface
SNMP CLI Netflow BGP NMS/EMS Netconf/YANG PCEP

Network(s)
WAN Orchestration Platform: Collector and Analytics
• Gathers topology, traffic, element, config, • Periodic view of current network model to
stats, etc. and places into the network build datastore for current and historical
database (current model) analysis

• Uses multiple feeds such as SNMP, CLI

parsing, Netflow, BGP-LS, etc
Collector Customer Analytics Costing Analytics
Analytics
Analytics Apps
Apps

Northbound APIs Analytics APIs

Optimization & Optimization &

Analytics Calendaring Analytics Calendaring
Prediction Prediction

Current Network New

Network New
Model
Current
Model Modeler Model Modeler Model

Network
Collector Deployer Network
Interface Collector Deployer
Interface
SNMP CLI Netflow BGP NMS/EMS Netconf/YANG PCEP
SNMP CLI Netflow BGP NMS/EMS Netconf/YANG PCEP
 WAN Orchestration Platform: Optimization and Prediction
Optimization &
Prediction

• Considers multiple criteria

including current and future Bandwidth Network Failure Apps
Placement Optimization Impact
demands, current model, global
vs tactical, etc.
OPM APIs

• Prediction
• pre-computes Optimization &
Analytics Calendaring
Prediction
optimal demand & path
placement under multiple failure Current Network New
scenarios on a per-interval or event Model Modeler Model

basis
Network
• querying “what if?” scenarios Collector
Interface
Deployer

SNMP CLI Netflow BGP NMS/EMS Netconf/YANG PCEP

 WAN Orchestration Platform: Calendaring Calendaring

• Schedules and reserves Apps Bandwidth Coordinated

Calendaring Maintenance
resources inside future window
of time
Calendaring APIs

• Reservations stored in persistent

database and considered in Analytics
Optimization &
Calendaring
Prediction
subsequent OPM and
Calendaring requests Current Network New
Model Modeler Model
• Requesting app could be notified
if reservation cannot be fulfilled Collector
Network
Deployer
Interface
• Not unlike getting bumped from a
SNMP CLI Netflow BGP NMS/EMS Netconf/YANG PCEP
flight
WAN Orchestration Platform: Network Modeler Network
Modeler

• Holds current and planned Apps Topology

Offline Simulation
Viewer
(simulated) state of the network
Network Model APIs
• OPM and/or Calendaring might
require deltas (changes) from
Optimization &
current and new model Analytics
Prediction
Calendaring

• Sequenced changes (model Current

Model
Network
Modeler
New
Model
staging) computed and passed
down to Deployer Collector
Network
Interface
Deployer

SNMP CLI Netflow BGP NMS/EMS Netconf/YANG PCEP

WAN Orchestration Platform: Deployer Deployer

• Programs networks elements

using different protocols and Apps
methods Northbound APIs

• CRUD data-paths across

single or multi-layer Analytics
Optimization &
Prediction
Calendaring
forwarding regimes
Current Network New
• Uses different software Model Modeler Model

components including Cisco

Network
Open SDN Controller, Cisco Collector
Interface
Deployer

NSO (Tail-F) and ODL, etc. SNMP CLI Netflow BGP NMS/EMS Netconf/YANG PCEP
Towards an SDN WAN
Orchestration Solution

WAE REST
APIs
Network Failure
Apps Planning Analysis Weather Map Trending

WAE Design (Planning) WAE Live (Analytics) Other Apps

Northbound APIs

WAN Automation Engine

Optimization &
Analytics Calendaring Analytics Calendaring
Prediction Optimization and Prediction

Current Network New

Model Modeler Model Current Model Network Modeler New Model

Network
Collector
Network
Deployer
Collector Deployer
Interface Interface

SNMP CLI Netflow BGP NMS/EMS Netconf/YANG PCEP SNMP CLI NetFlow BGP-LS NMS/EMS NETCONF/YANG EMS/NMS PCEP

SDN WAN Orch Platform (Architecture) Software Solutions (running code)

Comments on Software Solutions
• We have defined our SDN WAN Orchestration Platform Architecture and
functional components
• This architecture maps to a portfolio of SDN WAN Orchestration Software
Solutions (aka running code)
• Brief definitions follow of what these software solutions are and how they relate
to our SDN WAN Orchestration Platform
• Provide better context when discussing the solutions, showing running code and
listing the ingredients
• Actual configuration/deployment information will be covered in the Practical
Implementation Steps section
SDN WAN Orchestration Software Solutions – WAN
Automation Engine (WAE)
• Platform for all solutions; Operates as a server for all client applications
• Apps make use of some or all WAE platform components; basically a
prerequisite building block
• http://www.cisco.com/c/en/us/products/routers/wan-automation-
engine/index.html

WAN Automation Engine

Apps Analytics Optimization and Prediction Calendaring

Northbound APIs

Analytics
Optimization &
Prediction
Calendaring Current Model Network Modeler New Model
Current Network New
Model Modeler Model

Network
Collector
Network
Interface
Deployer Collector Deployer
SNMP CLI Netflow BGP NMS/EMS Netconf/YANG PCEP
Interface
SDN WAN Orchestration Software Solutions – WAE Design
• Application/UI for Design/Planning; Makes use of the WAE platform (server)
• http://www.cisco.com/c/en/us/products/routers/mate-design/index.html

Network Failure
Planning Analysis

WAE Design (Planning)

WAN Automation Engine

Apps

Northbound APIs Analytics Optimization and Prediction Calendaring

Optimization &
Analytics Calendaring
Prediction

Current Network New

Current Model Network Modeler New Model
Model Modeler Model

Collector
Network
Deployer Network
Interface Collector Deployer
SNMP CLI Netflow BGP NMS/EMS Netconf/YANG PCEP Interface
SDN WAN Orchestration Software Solutions – WAE Live
• Application/UI for Network Analytics; Makes use of the WAE platform (server)
• http://www.cisco.com/c/en/us/products/routers/mate-live/index.html

Weather Map Trending

WAE Live (Analytics)

WAN Automation Engine

Apps Analytics Optimization and Prediction Calendaring

Northbound APIs

Analytics
Optimization &
Prediction
Calendaring Current Model Network Modeler New Model
Current Network New
Model Modeler Model
Network
Collector Deployer
Collector
Network
Interface
Deployer Interface
SNMP CLI Netflow BGP NMS/EMS Netconf/YANG PCEP
WAN Orchestration
Solutions #1
Solutions Discussed in this Section
• Capacity Planning
• Failure Impact Analysis
• Multi-Layer Optimization
• Trending Analysis
• Health Analysis
• Coordinated Maintenance
WAE Design Dashboard
Demo
Solution: Capacity Planning

Management asking us “how much capacity

will we will need in the next 6 months?”
Solution: Capacity Planning
Architectural Context

Apps
① Design App 1
Design
② Analytics Module
③ Network Models 2 Analytics
Optimization &
Prediction

3 Current Network New

Model Modeler Model

Network
Collector
Interface
Solution: Capacity Planning

• Use WAE Design to

examine a model of the
network
• Apply a growth
percentage to demands
• Use the Create Growth
Plan tool to create
projected models
Solution: Capacity Planning Ingredients
• WAE Design
• WAE Platform
• Network Data  WAE (WAE collects network topology/traffic/stats from network
devices)
• jjjj

• networ

• N
Under Normal Conditions
• Everything is a-okay
• Network up, traffic flowing
smoothly …
• Plenty of Capacity
• No calls
• “.. I think we overpaid for this
gear ..”
Life is Good
But What Happens when …
• Something goes wrong!!
– Links (and nodes) down, Traffic
diverted, packets dropped
• Phone rings
• Management appears
• “..get the vendor on the Worst-Case Scenario. Life
phone!! STAT!!! ..” is NOT so Good

• Twitter lights up
Could we have predicted worst-case scenario
impact?
• Yes!!
• By a technique called Predictive Analytics
• Desired function of an SDN WAN Orchestration platform - consider past,
current and future loads to perform “what if” computations
• “what if scenarios” include:
• Normal and failure conditions under existing and anticipated traffic loads
• Failure conditions under existing loads
• Growth Scenarios
Solution: Failure Impact Analysis

Need to understand where traffic will go

and what the impact will be if something
changes on the network (planned or
unplanned).
Solution: Failure Impact Analysis
Architectural Context

Apps
1
① Design App Design

② OPM
Optimization &
③ Network Models Analytics
Prediction 2

Current Network New

3 Model
Model Modeler

Network
Collector
Interface
Solution: Failure Impact Analysis
Single Failure
• Use WAE Design to
visualize the network
utilization
• Identify how traffic
traverses the network
• Simulate failures or
maintenance plans and
examine where traffic will
go and what the impact
will be
Solution: Failure Impact Analysis
Simulation Analysis
Automates the process of
examining a large set of
failure scenarios
Failure Sets:
Circuits, Nodes, Sites, L1 Links, L1
nodes, srlgs, ports, port circuits
Solution: Failure Impact Analysis
Simulation Analysis
Worst-Case Traffic Failure Impact
• Interfaces most at risk • Failures that will cause congestion
somewhere else
 Reference

Solution: Failure Impact Analysis Ingredients

• WAE Design
• WAE Platform
• Network Data  WAE
 Reference

Layer 3/Layer 1 Modeling

• Layer 3 and layer 1 contained in the same model.
• Useful for failure analysis: Fail a link or node at layer 1 and see the impact at layer 3
• Supported Vendors
• Ciena, Infinera, Cisco

• Collected Data
• Basic L1 topology: L1 nodes, , L1 Links, L1 Circuits, L1 Circuit Paths
• Cisco: Wavelengths, feasibility metrics, SRLG’s and setup priorities

• Model Building
• Producing ROADM-to-ROADM links
• Waypoints
Solution: Multi-Layer Optimization

I need to understand how a failure to my

layer 1 topology can affect my layer 3
network. How can I optimize my layer 3
topology?
Solution: Multi-Layer Optimization
Architectural Context
Apps 1
Design

① ML Design Application ML OPT NW Cost

Add-On Add-On 2 3
② Optimization (e.g. Add-on for Optimization &
Prediction 4
Multi-Layer Optimization)
Current Network New
③ Costs (e.g. Add-on for Network Model Modeler Model 5
Cost Calculator)
Network
Collector
Interface
④ OPM
⑤ Multi-Layer Network Models
L3 – IP/MPLS

L1 - Optical
Solution: Multi-Layer Optimization
Failure Impact Analysis
• The WAE Plan file allows you to represent a layer 1 topology together with a
layer 3 topology.
• WAE provides the mapping capability between the two layers, allowing you to
examine how failures at layer 1 can impact the layer 3 topology.
Solution: Multi-Layer Optimization
Multi-Layer Design Add-on

• For areas of traffic growth, this add-on determines where to add layer 3
circuits and corresponding optical circuit paths
• This is to achieve the required utilization levels under normal and failure
conditions
• Objective is cost optimization
Solution: Multi-Layer Optimization
Network Cost Calculator Add-on
• This add-on can create a bill of materials, and provide a before and after view of
cost savings
Multi-Layer Optimization
Demo
Solution: Multi-Layer Optimization
Ingredients
• WAE Design
• WAE Platform
• Layer 1 information gathering from Ciena, Infinera and Cisco
• Collected data
• Basic L1 topology: L1 nodes, , L1 Links, L1 Circuits, L1 Circuit Paths
• Cisco: Wavelengths, feasibility metrics, SRLG’s and setup priorities

• Model building
• Waypoints and ROADM-to-ROADM links
• Network cost modeling and network cost calculator
Quick Notes on Add-ons and CL Tools (1) Reference

Add-ons: Additional software functions that run in WAE Design UI

• Double-Failure SLRGs
• Line Card SRLGs
• Circuit Maintenance
• Config Generator
• LSP Config Update
• Bundle Parallel Circuits
• L1 Topo Import; L1 Circuit Import
• L1 Export
• Trim Nodes Whitelist
• Power Analyzer
Quick Notes on Add-ons and CL Tools (2) Reference

CL Tools: Command Line (CL) functions that run on the WAE Platform
• Many used to enhance network model contents
• BGP Endpoint Init • Remove_interfaces

• P2MP LSP Init • Peering_links_report

• Ext_endpoints_tags_creator • Merge_topologies

• Ext_endpoint_init • Model_accuracy_meter

• Int_threshold_report • Power_Collector

• Collect_inventory

• Infinera_import

• Ciena_import_from_db
Traffic Trending
• Need to understand how traffic
has changed on my network over
time.
• Using time-series data, WAE
Analytics can generate a report
traffic on my network and
compute traffic trends
• Examples:
• Per site or site to site

• Based on traffic of interfaces,

LSPs
Solution: Trending Analysis

How can I understand traffic patterns on my

network over the past 6 weeks or 6
months? And then estimate traffic patterns
for the next 6 weeks or 6 months?
Solution: Trending Analysis
Architectural Context

1 Analytics Apps
① Analytics App
② Analytics API 2 Northbound APIs

③ Analytics Module 3 Analytics

④ Network Models Current Network New

4 Model Modeler Model

Network
Interface
 Solution: Trending Analysis Workflow
• Requirement is a report
of network utilization
• Create report in WAE
Analytics application,
selecting evaluation
criteria, time range,
projection

• Schedule the report to run and email the results

Solution: Network Health Analysis

I need better network performance data.

I want reports that identify network “health”
issues. Things like LSPs that change paths too
often or routers exceeding their CPU/memory
thresholds.
Solution: Network Health Analysis
Architectural Context

① Analytics App 1 Analytics Apps

② Analytics API 2 Northbound APIs

③ Analytics Module
3 Analytics
④ Network Models
Current Network New
4 Model Modeler Model

Network
Interface
Solution: Network Health Analysis Workflow
• Requirement is a
historical analysis of my
network to understand
health issues
• Create report in WAE
Analytics application
• Example Find the LSPs
that changed paths the
most
Network Health Analysis
Video
 Reference
Solution: Network Trending and
Health Analysis Ingredients
• WAE Live
• WAE Platform
• Network Data  WAE
Maintenance Downtime
• Need to be able to schedule
maintenance events, and
know in advance what the
impact to the network will be
• Things to consider:
• May not need to wait for off-
peak hours
• May be able to schedule
multiple maintenance events
at the same time

How many times have we heard about a network outage due to

some network upgrade?
Solution: Coordinated Maintenance

What is the impact on my network if I need

to take a few routers (or interfaces) out of
service for maintenance?
Solution: Coordinated Maintenance Solution
Architectural Context

Apps Coordinated Maintenance 1

① Coord Maint App
Northbound APIs 2
② Calendar API
③ Calendar Module 4 Analytics Calendaring 3

④ Analytics Module Current

Model
Network
Modeler 5
New
Model

⑤ Network Models Network

Collector Deployer
Interface
Solution: Coordinated Maintenance
Solution Workflow
• Maintenance required; need to determine best window
• Select resources (e.g. node, node i/f, etc.)
• Select Maintenance Window Start Time and Duration to system for
evaluation
• Platform responds with scheduling conflicts, network impact (e.g.
congestion)
• Proceed with Coordinated Maintenance Scheduling
 Coordinated
Maintenance Demo
Use-Case: Coordinated Maintenance Ingredients

• Coordinated Maintenance Application

• WAE Platform
• Network Data  WAE
Summary & Part2 Setup
Conclusions So Far (1)
• Network Operators continue to face challenges along multiple fronts
• Bandwidth use rising, demands changing rapidly, network now a utility
• Need to lower costs, drive additional services

• Operator knowledge/experience + TE disciplines enhanced with new

software platforms and applications
• We have explained and demonstrated several application/use-
cases/solutions supporting network planning, trending and analysis
• Note that the solutions discussed so far DO NOT include any device
programming
Conclusions So Far (2)
• Note that the solutions discussed so far DO NOT include any device
programming
• But they all subscribe to the notion laid out in our SDN definition

• In the next section we will discuss

solutions that include device
programming using different
protocols
The TE Timeline … Reference
Start Here with PCE

2015
What next?
• We have the Stateful PCE notion and architecture defined. And certainly interest
in deployment across MPLS and GMPLS networks
• But this is just a functional subset of what is required in a WAN Orchestration
platform
• Other areas to consider:
• Applications (old and new ones)
• Northbound APIs based on REST
• Advanced optimization, calendaring, analytics, etc. processes WAN Value-add

• Southbound protocol flexibility (for collection and programming)

• Device Programming

It is now all about the applications

Inflection Point of New Technologies?
• Traffic modeling has matured
• Predictive Analytics and Simulation
• Rise of Open Source Software
• SDN Controllers == Application Development Platforms
• Focus on Applications and APIs (Java, Python, DB, REST, HTML5/Javascript, etc.)
• Resource elasticity and virtualization enabled by NfV
• Interoperable network collection and programming standards
• Finally doing something with all of that collected network data
• What else?
Venn Observation of SDN WAN Orchestration
Software
Engineering

Cloud
Where NG
Operators Need
Tail-F NCS
to Play

Stateful PCE
MPLS TE

ISIS
Segment
Routing
BGP Network
Engineering
Break
Agenda – Part 2
• Protocol & API Primer
• WAN Orchestration Solutions #2
• Bandwidth on Demand
• Bandwidth Scheduling
• Minimize Congestion in a Segment Routed Network
• Tunnel Split/Merge
• Tunnel Balancer
• Intelligent Workload Placement
• Reactive
• Practical Implementation Steps & What You Need to Know
• Summary & References
Protocol & API Primer
Brief Remarks (1)

• Should NOT care anymore about which protocol does what

• Why is that?
• Focus on needs and business outcome located at the workflow,
orchestration, application and API layer
• SDN platforms “abstract away” all of the protocol details
• We have other non-protocol methods to configure/program network
elements
• Networks are heterogeneous, we just use what works
Brief Remarks (2)
• But as network engineers, architects, planners … the
people that have built, operated and managed networks, we
need to know a bit more
• Because:
• Platform implementation differences
• Network elements lack specific protocol support
• Always good to be up-to-date on what’s happening
SDN WAN Orchestration GPS - Protocols
Apps

Northbound APIs

Optimization &
Analytics Calendaring
Prediction

Current Network New

Model Modeler Model

Network
Collector Deployer
Interface

Protocols SNMP CLI Netflow BGP NMS/EMS Netconf/YANG PCEP


Brief Run Thru of the following …
Key Function
IGP Topology
Create, Modify and Delete TE
or SR Tunnels
Classification and Action
Read/Write of Persistent
Configuration Data on Network
Devices
Reference
Protocol/API Comments
BGP Link-State Wraps up LSDB in BGP transport and pushes to BGP
speaker on SDN WAN Orch Platform
Stateful Extensions to Introduced as part of Stateful PCE effort
PCEP
Openflow Extensions Leveraging per-flow MATCH/Action semantics
Netconf/Yang (Demo) Open source and vendor implementations; now being
deployed
Use BGP to Push Link State Database to
SDN WAN Platform (Apps)

• BGP Link-State (BGP-LS) advertises

network link state and link state TE up SDN WAN Orchestration
to SDN WAN platform
BGP-LS
• Familiar BGP operational model
• Need at minimum single BGP-LS
speaker per domain IGP LSDB carried in
BGP
• Implementations
• WAE, ODL, Open SDN Controller, IOS-
XR Network

BGP-LS
Speaker
Path Computation Element Protocol (PCEP)
Path/TE App

• Convey Tunnel information between

SDN WAN Orch platform (aka PCE in PCEP
parlance) and routers SDN WAN Orchestration

• PCEP sessions (TCP) between PCE and PCEP

router
• Two current modes based on Stateful PCEP PCEP
PCE initiative:
• PCE Initiated: APP + PCE initiate tunnel
setup PCEP

• LSP Delegation: router initiates tunnel setup Tunnel

(e.g. via CLI or NMS) then delegates tunnel Edge Routers
management to PCE Head-End WAN
 Deployed PCEP LSP Delegation Example
Path/TE App

① LSPs (tunnels) are first created by

operator using operator/NMS using
NMS
CLI/XML
SDN WAN Orchestration

PCEP
② Operator delegates/un-delegates a
configured tunnel via CLI. PCErpt
message sent to PCE-A (and PCE-
B if present)
1 2
③ Now only PCE can control the
LSPs
4
PCEP
④ If PCEP session goes down and
LSP
does not come backup within a
Edge Routers specified time, and if there is no
Head-End WAN backup PCE-B, routers regain the
control of delegated LSPs
SDN WAN Orchestration and Openflow
• Not likely, near-term, to program per-flow state across a contiguous set of
Openflow-enabled WAN elements
• for scale, resiliency, service enablement, controller overhead, etc. reasons

• But per-flow state does exist at the edge of the WAN

• that is usually handled via CLI-configuration
• Example is policy-based routing (PBR) for flow indirection

• Openflow possesses ability to program flow entries in a flow table

• Definition of a flow entry
• “.. element in a flow table used to match and process packets. It contains a set of
match fields for matching packets, a priority for matching precedence, a set of
counters to track packets, and a set of instructions to apply..”

97
SDN WAN Orchestration and Openflow for Traffic
Steering Traffic Steering
App

• Use Openflow to program

classifiers/actions on WAN Edge
SDN WAN Orchestration

• Flow entries would be something

OF PCEP
like:
Flow Entries
• MATCH/Set_IPv4_Next_Hop
<MATCH/”Forward to
• MATCH/Set_IPv6_Next_Hop Tunnel”>
• MATCH/Set_FCID (tunnel identifier) Create LSP Tunnel
• MATCH/Set_VRF

• Useful for services and applications OF PCEP

requiring steering of specific flows Flows Tunnel
into a programmed WAN resource Edge Routers
Head-End WAN

98
 Reference

Brief History of Netconf/YANG

• SNMP and CLI have been around forever
• Overview of the 2002 IAB Network Management Workshop defined Operator
Requirements
• Source: RFC3535
• Netconf developed (2006) to read/write configuration data between client (e.g.
NMS) and server (e.g. router)
• Initially content-agnostic, needed a data model
• YANG developed (2010) as data model language for Netconf
• XML-based, human-readable, flexible and extensible
 Reference

Netconf/YANG

• Emerging method for:

• configuring network devices and NMS/OSS
services YAN
G
• Receiving notifications “pushed”
from network
Read/Write
Notifications
Config Data

YAN YAN
G G

NC NC NC NC

Routers YAN YAN Routers

G WAN G

100
 Reference

Netconf

Source: RFC6241
 Reference
What is YANG?
• Data modeling language for configuration and operational objects that can be
manipulated on the network device
• Via NMS, SDN App/Controller, Orchestration platform

Router module
Header information
Imports & Includes
properties container
Type definitions
name: string, config
Configuration & Operational
data declarations
interfaces container

Router Action (RPC) & Notification declarations

interface: list, index = name

name: string, config

oper-state: enum, config

www.sublimetext.com/2
https://github.com/apackeer/sublime-yang-syntax
https://github.com/mbj4668/pyang

YANG Tools Demo

Why is YANG Important?
• Becoming the defacto industry standard for service and device data
models
• SDN Apps/Controller Platforms are becoming model-driven
• Model-Driven REST APIs  RESTCONF
• SDN WAN Orchestration Platform – Netconf/YANG
SDN WAN Orchestration GPS - APIs
Apps

APIs Northbound APIs

Optimization &
Analytics Calendaring
Prediction

Current Network New

Model Modeler Model

Network
Collector Deployer
Interface
SNMP CLI Netflow BGP NMS/EMS Netconf/YANG PCEP
 What is REST?
• REpresentational State Transfer (REST) is an API architecture style that can use HTTP to
send messages between a client and server, piggyback on existing HTTP Internet

Applications
HTTP Packets
Perl

Python

Server
App
REST API Server
C#

Java

• Applications can Create/Read/Update/Delete

• REST services are platform-independent, language-independent
• App can be browser-based or standalone
REST API Example: Bandwidth Calendar Reference

PUT HTTP://SDN.WAN.PLATFORM/BANDWIDTH-CAL-APP/ HTTP1.1

NB API

Request URL: HTTP://SDN.WAN.PLATFORM/BANDWIDTH-CAL-APP/

SDN WAN Orchestration
Request Method: POST
Status Code: 200 OK
Request Headers
Content-Type: application/json
IPv4/IPv6/MPLS/SR
Host: SDN.WAN.PLATFORM
Request Payload
Optical
{"requests":[{"src":”R1","dst":”R2","bandwidth":”5000”, “strt”:”2200”,
“end”:”0300”, “Cust”:”Acme:}]}

107
WAE Platform API Categories
APIs Description
• Topology Nodes, Interfaces, Sites, LSPs
• Demands Transient, Durable and Calendared
• Analytics Time-series reporting
• Optimizations Path, Tactical & Explicit optimization

• Network Collection Unprocessed collected data

• Model Low-level model manipulation
• System Management Plan processing and staging
WAE Platform API – Query and Admit
• Demand and Optimization define Query and Admit APIs
• Query asks “What if?”
• Admit changes the network model
Example Optimization Query and Admit APIs
• Same JSON contents
• Different URL

Snippet from https://developer.cisco.com/site/wae/documentation/reference-guide/

https://developer.cisco.com/site/wae/

Quick Devnet/WAE API

Demo
 Reference

RESTCONF Overview
• Requirement: WEB Apps access to
configuration and operations data on
network devices
• Definition: “.. a REST-like protocol …
running over HTTP [RFC2616], for
accessing data defined in YANG [RFC6020],
using datastores defined in NETCONF
[RFC6241] ..”
• draft-ietf-netconf-restconf
• Implementation
• OpenDaylight
Source: draft-ietf-netconf-restconf
 Reference

Netconf vs RESTCONF

WEB UI, Python, …

GET, PUT, POST …;

JSON/XML-Encoding

HTTP
 Reference
RESTCONF API (1) - ODL
RESTCONF API (2) – Reference

ODL

HTTP Operation

API Doc

URL and Response

WAN Orchestration
Solutions #2
Solutions Discussed in this Section
• Bandwidth on Demand
• Bandwidth Scheduling
• Tunnel Split/Merge
• Tunnel Balancer
• Intelligent Workload Placement
• Reactive
Traffic Demands
• Demands define the amount of data transmitted between each pair of network
nodes
• Types of traffic demands supported in SDN WAN Orchestration (WAE)

Type Description Example App

Transient Submitted to system and accounted for until the Bandwidth on Demand
next data collection
Persistent Submitted to system, stored in a database, Bandwidth Scheduling
accounted for up until and including its
scheduled start-time and duration
Durable Submitted to system, stored in database and Any app computing new network
accounted for until it is deleted model where traffic demands are
accounted for
Solution: Bandwidth on Demand

I have a request for immediate bandwidth.

Can my network support it?
Solution: Bandwidth On Demand
Architectural Context

Apps Bandwidth on Demand 1

① Bandwidth on Demand App
Northbound APIs 2
② Transient Demand API
Optimization & 3
③ Optimization and Prediction Prediction
Module
Current Network New
Model Modeler
4 Model
④ Network Models
Network
⑤ Deployer Interface
Deployer 5
 Solution: Bandwidth on Demand Workflow
• I need to support
immediate BW on my
network
• Select source,
destination and amount
of bandwidth
• Platform evaluates
network impact
• If not enough capacity, request optimization
• If acceptable, proceed with bandwidth request
Bandwidth on Demand
Demo
Solution: Bandwidth on Demand Ingredients
• Bandwidth on Demand App
• Bandwidth On Demand API
• WAE Platform
• Network Data  WAE
• Network Devices  WAE (program network device(s))
• If using configuration: use Cisco NSO (Tail-F)
• If using protocol PCEP: OSC and XR 5.3.1 if using PCEP
Solution: Bandwidth Scheduling

Can I sell bandwidth on my network, taking

advantage of times when the network is
under-utilized? Maybe price it accordingly.
Solution: Bandwidth Scheduling
Architectural Context

Apps Bandwidth Scheduling 1

① Bandwidth Scheduling
Northbound APIs 2
Application
② Calendar API Calendaring 3
③ Calendar Module Current Network New
Model Modeler Model 4
④ Network Models
Network 5
⑤ Deployer Interface
Deployer
Solution: Bandwidth Scheduling Workflow
• Customer wants to
purchase bandwidth for
given time period; need
to request system to
evaluate impact
• Platform evaluates
network impact
(considers traffic, other
BW requests)
• App can consult with other business systems
• If price is acceptable customer can schedule bandwidth, reservation created
in WAE
Bandwidth Scheduling
Demo
Solution: Bandwidth Scheduling Ingredients
• Bandwidth Scheduling Application
• WAE Platform
• Network Data  WAE
• Network Devices  WAE
• If using configuration: use Cisco NSO (Tail-F)
• If using protocol PCEP: OSC and XR 5.3.1 if using PCEP
 Reference

Segment Routing Overview

• Simple routing extensions to implement source routing
• Packet path determined by prepended segment identifiers (one or more)
• Reduced network state
• Increased network scalability and virtualization
• Data-plane agnostic
• MPLS: label as segment id
• IPv6: address as segment id

• The state is no longer in the network but in the packet

• datatracker.ietf.org/wg/spring/documents/
 Reference
Segment Routing TE
• Same Look/Feel, features (e.g.
SR-TE RSVP-TE auto b/w) as RSVP-TE tunnels
TE state only at head-end Yes No
SDN WAN Orchestration

PCEP
SR-TE tunnel
Engineered for SDN Yes No • Provisioning Options:
ECMP-capability for TE Yes No
16001 • Configurable (CLI/XML); PCEP (App
+ PCE or Delegated); Netconf/YANG
16005 Coming
Source Destination
• Less complexity and higher scale:
16001
16005
• Encourages operator to deploy TE
Data solution
https://datatracker.ietf.org/doc/draft-ietf-pce-segment-routing/ • Opens the possibility of innovating
new use-cases
Solution: Minimize Congestion in a Segment Routed
Network

We are required to meet SLAs for a specific

customer. This involves identifying network
“hotspots” and minimizing their impact on our
important customer. Our network recently
transitioned to Segment Routing.
Solution: Minimize Congestion in an
SR Network
Architectural Context
Apps Path Analysis/Provisioning 1
① Path A/P App
Northbound APIs 2
② OPM API
Optimization & 3
③ Optimization And Prediction Prediction
Module
Current Network New
Model Modeler
4 Model
④ Network Models
Network 5
⑤ Deployer Interface
Deployer

Segment Routing
Solution: Minimize Congestion in an SR
Network
• Retrieve current model from WAE platform using REST API
• Identify network “hotspots”
• Run SR Congestion Avoidance Add-on – yields optimal SRTE path
• If acceptable, provision SRTE paths
Minimize SR Network
Congestion Demo
Solution: Minimize Congestion in an SR Network
Ingredients
• WAE Design + SR Congestion Avoidance Add-on
• WAE APIs
• WAE Platform
• Segment Routing Enabled Network
• Network Data  WAE
• Network Devices  WAE
• Using protocol PCEP: OSC and XR 5.3.2 that supports SR
Solution: Tunnel Split/Merge

We have an auto-bandwidth network. We want to

make sure my LSPs don’t become too large (e.g.
exceed link bandwidth, unused space capacity).
What about if we have too many smaller LSPs
that could easily be combined into one larger
LSP?
Solution: Tunnel Split/Merge
Architectural Context

Apps Tunnel Split/Merge 1

① Tunnel Split/Merge App
Northbound APIs 2
② OPM API
Optimization & 3
③ Optimization And Prediction Prediction
Module (OPM) Current Network New
Model Modeler
4 Model
④ Network Models
Network 5
Deployer
⑤ Deployer Interface
Solution: Tunnel Split/Merge Workflow
• Tunnel
Split/Merge App
examines user-
specified global
Max and Min
Setup BW
thresholds.

• LSP
programming
• Tunnel Split: If LSP > Max Threshold
• Tunnel Merge: If LSP < Min Threshold and there is more than 1 LSP with the same
source/destination node.
Tunnel Split/Merge
Manager Demo
Solution: Tunnel Split/Merge Ingredients
• Tunnel Split/Merge App
• WAE Platform
• Network Data  WAE
• Network Devices  WAE
• PCEP: OSC and XR 5.3.1 if using PCEP
Solution: Tunnel Balancer

We are a global SP and operate expensive

transoceanic links. NOC says we need to
react quickly to balance traffic in LSPs. We
are told this will save $$ and make for happy
customers.
Solution: Tunnel Balancer
Architectural Context

Apps Tunnel Balancer 1

① Tunnel Balancer App
② OPM APIs Northbound APIs 2

③ Optimization And Prediction Optimization & 3

Module Prediction

Current Network New

④ Network Models Model Modeler
4 Model

⑤ Deployer (future) Network

Deployer 5
Interface
Solution: Tunnel Balancer Workflow
• Simulate a failure (circuit,
fiber cut)
• Observe impact to network
• Use WAE to balance the
loadshare values on the
LSPs
Current
• Observer LSP loadshare
changes necessary

Rebalanced
 Solution: Tunnel Balancer Ingredients

• WAE Design
• WAE Platform
• Network Data  WAE
• Future
• Deployer support for re-balancing LSPs
Brief Interlude from WAN Orchestration Solutions to Discuss

• SDN WAN Orchestration and Netconf/YANG

• Cisco NSO as a Deployer Plug-in
• LSP YANG Model
 Reference
SDN WAN Orchestration and Netconf/YANG

• Emerging method for

• configuring network devices and
services SDN WAN Orchestration

• Receiving notifications “pushed” from YANG

Netconf
network
• Incorporated into southbound-
facing components without Read/Write
Config Data
Notifications
impacting higher layer WAN
Orchestration logic or applications
YANG YANG

NC NC NC NC

Routers YANG YANG Routers

WAN

145
Cisco Network Services Orchestrator (NSO) Reference
formerly known as Tail-f NCS

Management Network
Applications Engineer • Logically centralized network
services
REST, NETCONF, Java, etc. Networkwide CLI and Web UI

Tail-f NCS • Data models for data structures

Service
Models
Service Manager
• Structure representations of:
Device Manager • Service instances
Device
Network Element Drivers Models
• Network configuration and state

• Mapping service operations to

NETCONF, CLI, SNMP, REST, etc.
network configuration changes
• Execution
management
• Transactional integrity
systems (EMSs)
• Applications
• Controllers • Multiprotocol support
• Multivendor support
SDN WAN Orchestration
Platform + NSO Apps
Architecture Northbound APIs

• WAE will use the NSO Analytics

Optimization &
Calendaring
Prediction
NETCONF native interface
Current Network New
• NSO is a separate installation, Model
Model Modeler
will need to load any required
Service Templates and NED Collector Network Deployer
updates Interface NSO Plug-in

• Perform LSP create, read, LSP CRUD

update and delete operations
NSO WAE Service

• NSO uses Netconf/CLI to Models

Service Manager

configure routers Device Manager

WAE Device
Models

Network Element Drivers

Netconf, CLI
Quick Look at the LSP
YANG Model
Orchestration – Then and Now

OSS/BSS OSS/BSS

REST, Netconf, …

CPE WAN DC Orchestration YANG

Prov Prov Prov Platform
Service
Models

CPE WAN Data Center

CPE WAN Data Center

YANG
Device
Models

Customized BSS/OSS and

Provisioning Systems Model-driven Network and Service
Orchestration
Orchestration and SDN Apps/Controller Working
Together (1)
OSS/BSS/Portal/etc

• Network/Service REST, Netconf, …

Orchestrator talks to
per-domain SDN Orchestration
Apps/Controller Platform

REST, RESTCONF …
Netconf, CLI …

• Objective: Establish SDN Apps/ SDN Apps/

and Manage e2e Controller Controller

service
CPE
Data Center
WAN
How Does WAE Fit into the Picture?
Orchestration Orchestration
Platform Platform

Opt/Planning/Cal API Opt/Planning/Cal API

BGP/PCEP/Netconf/etc Orchestration (Netconf/CLI)

Netconf, CLI

WAN WAN
Need for X-Domain Orchestrated Services
X-Domain Service Orchestration/
App/Portal

Cust Data Center

WAN Data Center

E2E Service
Solution: Intelligent Workload Placement

We are a provider of cloud hosting

services. We need a way to figure out
where to place and configure workloads in
a data center so that the data center *AND*
WAN resources are used most efficiently.
Solution: Intelligent Workload Placement
Intelligently place workload into a data center within
my network so I can optimize load placement. App that queries both systems and produces a set of
Intelligent options to optimize load placement that
guarantee
Workload Placement App SLA, allocate sufficient resources, optimize
network utilization, minimize latency

WAN Automation Engine 1. User Registers Workload DC Resource Manager

Requirements
Worst Case Latency Amount of CPU
2. App queries both systems
Worst Case Utilization Amount of Memory
3. Computes different load
Network Bandwidth
placement options based on Amount of Disk
Policy Workload
SLAs, available resources,
network % and latency
4. Operator assesses and selects
Intelligent Workload Placement Options

Workload
Requirement
Solution: Intelligent Workload Placement
Ingredients

• Intelligent Workload Placement App/Portal

• WAE Platform
• DC Resource Manager
• WAE and DC Resource API
Solution: Reactive Control

How can my network react to conditions

without human involvement?
Solution: Reactive Control
Architectural Context

Apps Policy Definition 1

① Policy Definition App
Northbound APIs 2
② Policy API
Optimization & 3
③ Optimization and Predictive Prediction
Analysis Module
Current Network New
Modeler
4 Model
④ Plan module Model

Network
⑤ Network event driven collection 5 Collector Deployer 6
Interface
⑥ Deployer (Segment Routing)

Segment Routing
Solution: Reactive Control

• An application defines a policy in

WAE
• A policy can examine SR paths for
lowest delay, igp or TE metric value
and consider elements of the topology
to avoid
• Based on event driven collection,
WAE can rapidly detect and react to
changing network conditions
• Policy management is handled by
WAE
Solution: Reactive Control
Ingredients
• Policy App/UI
• WAE Policy Management
• Network Data  WAE
• Network Device(s)  WAE
• Using protocol PCEP: OSC and XR 5.3.2 that supports SR
• Enable SR on the network
Practical Implementation
Upfront Practical Implementation Remarks
• Implementing and deploying an SDN WAN Orchestration solution based on the
WAE platform can vary depending on multiple factors
• Particular problem; which app, who will develop/test, involve network programming, etc.

• Focus on two primary implementation task frameworks

• Planning
• WAE Install/Config/Deployment

• Reminder that there are multiple options for execution of a successful

deployment
Practical Implementation Task Roadmap
Planning

Identify Map out Solution Solution Solution Rollout

Problem Flow Implementers Planning

WAE

WAE App Rollout

Sys Config/ App
Install Collector Model Development/T
Building esting

Deployer
Planning Identify
Problem
Map out Solution
Flow
Solution
Implementers
Solution Rollout
Planning

• Problem and Solution

• Covered today or something else?

• Map out solution flow or steps – e.g. flow diagram/chart

• Solution Implementers
• Existing staff
• Contractors under staff guidance
• Cisco Application Programming Group (APG)
• Cisco Network Modeling & Business Intelligence Services (NMBIS)

• Solution Rollout Planning

• Include testing, limited/phased deployment, etc.
 Cisco Network Modeling & Business Reference

Intelligence Services (NMBIS) Planning

Identify
Problem
Map out Solution
Flow
Solution
Implementers
Solution Rollout
Planning

WAE

Cisco Network Modeling & Business Intelligence Install Collector

Sys Config/
Model
App
Development/
WAE App Rollout

Services at a Glance Building Testing

Deployer

Network Peering
Failure & Custom BI Dashboard Architecture
Topology & Optimization,
Change Impact Capacity Analytics Assessment &
Behavioral Planning &
Analysis Planning Customization Design
Analysis Cost Analysis

Assesses
current network
Evaluates core
Assesses
network ability
Assesses
network
Provides
standard and
Evaluates and
recommends state against Additional WAN Orchestration Related Services
infrastructure that business &
to support readiness to customized actions to
supports mission- technical
current loads support future dashboard optimize
critical requirements
and handle traffic growth or views to suit analytics and
applications and and proposes
potential failure business your business network design Installation & Customization
services
conditions initiatives model changes design for WAN Application
Optimization & System
Orchestration/ Development *
Optimization for WAE Integration

© 2013 Cisco and/or its affiliates. All rights reserved. 3

Custom application
development to
Planning, installation, Customization of interface with WAE
optimized WAE platforms and
configuration, and integration with • Currently evaluated
knowledge transfer orchestration and and offered on a
for WAE suite backend systems case-by-case basis

© 2013 Cisco and/or its affiliates. All rights reserved. 4

WAE
Overview
• WAE Platform runs on a server that has reachability to the network
• WAE Applications interact with the WAE Platform using APIs.
• Applications such as WAE Live and Coordinated Maintenance are accessed using a Web Browser
• WAE Design runs on Windows, Mac or Linux. It is used offline with plan files from the WAE Platform or can be
used independently for greenfield planning

WAE REST
APIs
WAE Applications Network Failure
Planning Analysis Weather Map Trending

WAE Design (Planning) WAE Live (Analytics) Other Apps

WAN Automation Engine

Analytics Optimization and Prediction Calendaring

WAE Platform Current Model Network Modeler New Model

Network
Collector Deployer
Interface

SNMP CLI NetFlow BGP-LS NMS/EMS NETCONF/YANG EMS/NMS PCEP

WAE
Platform Installation and System Sys Config/ App
WAE App Rollout
Install Collector Model Development/

Configuration Building Testing

Deployer

• RHEL 6.5 with minimum system requirements

• Installation is RPM based.
• Requires root/sudo privileges
• Requires access to repositories to download standard Linux packages not included in the
WAE distribution.

• If using HA, you may need to deploy a number of hosts

• Also need ssh-keys between hosts for system file transfer

• Need to open ports for web interfaces

• Adjust the number of CPUs and memory used by the WAE Platform (Default: 8
CPUs, 32 GB RAM)
More on WAE High Availability

• Northbound HA
• One virtual IP shared by keep Northbound (Cluster)* Northbound (Cluster)
alive cluster

• Core HA
• Active/Standby configuration

• PCEP HA WAE Platform (Active) WAE Platform (Standby)

WAE Platform (Active)
• Edit the CDL configurations to
Distributed Distributed
point to WAE as the primary JMS DB** DB
broker
• LSP states are kept in sync with
Standby PCE by PCC on the
router OSC (Cluster***) OSC (Cluster)

* Group of computing resources viewed as single resource. Example above

handles API requests.
** Cassandra DB
*** Cisco value-add
 WAE Install Collector
Sys Config/
Model
Building
App
Development/
Testing
WAE App Rollout

Collector Configuration Deployer

• Collection from the network requires the following information:

• Seed router, backup seed router, IGP type, area/level, telnet or ssh, snmp community string and
login details

• Collector is configurable using a web-interface or cli

• Web-interface collection is not as flexible as cli based snapshots, but easier to troubleshoot and
maintain

• Additional required information (if necessary)

• Specific nodes to avoid or include WAE REST
APIs
• Router management addresses if the IGP loopback addresses are Network
Planning
Failure
Analysis Weather Map Trending

not reachable WAE Design (Planning) WAE Live (Analytics) Other Apps

• Choose what to collect and poll and the frequency WAN Automation Engine

• Interfaces, RSVP LSPs, QoS, Multicast Analytics Optimization and Prediction Calendaring

Current Model Network Modeler New Model

Network
Collector Deployer
Interface

SNMP CLI NetFlow BGP-LS NMS/EMS NETCONF/YANG EMS/NMS PCEP

WAE Install Collector
Sys Config/
Model
Building
App
Development/
Testing
WAE App Rollout

Network Modeler Deployer

• Building a reasonable demand mesh*

• Need to choose appropriate demand endpoints
• Build a demand mesh per service class
• Need to make sure failure scenarios are properly
modeled

• Demand Deduction – use network

measurements to determine demand
traffic values WAE REST
APIs
Network Failure
Planning Analysis Weather Map Trending

WAE Design (Planning) WAE Live (Analytics) Other Apps

WAN Automation Engine

Analytics Optimization and Prediction Calendaring

Current Model Network Modeler New Model

* Partial/Full mesh of end-points supporting
various traffic demands Collector
Network
Deployer
Interface

SNMP CLI NetFlow BGP-LS NMS/EMS NETCONF/YANG EMS/NMS PCEP

 WAE Install Collector
Sys Config/
Model
App
Development/
WAE App Rollout

App Development (& APIs) Building Testing

Deployer

• Applications can be started on the same server

as the WAE Platform
• To build other applications, it’s recommended to
test the workflow in WAE Design using a model
of your network
• APIs:
Apps
• Understand Demand and Optimization and
how/when they are used Northbound APIs

• Understand Query and Admit APIs WAN Automation Engine

• Use Staging APIs if there will be more than one Analytics Optimization and Prediction Calendaring
application using the WAE Platform
Current Model Network Modeler New Model

Network
Collector Deployer
Interface
 Reference
More on WAE APIs
• Lab 1: Introduction to the WAN Automation Engine (WAE) REST API
• Example Application in Python - explore the network topology
• https://learninglabs.cisco.com/#/labs/wae-1/step/1

• Lab 2: Lab 2: WAE Predictive Analysis for Demand Admission

• Example Application in Python - determine the total available bandwidth between nodes in normal and worst-case
• https://learninglabs.cisco.com/#/labs/wae-2/step/1

• Lab 3: Basic LSP Operations with WAE

• Example Application in Python - Tunnel Split
• https://learninglabs.cisco.com/#/labs/wae-3/step/1

• Lab 4: WAE LSP Optimization

• https://learninglabs.cisco.com/#/labs/wae-4/step/1

• API guide: https://developer.cisco.com/site/wae/documentation/reference-guide/

• Devnet WAE - https://developer.cisco.com/site/wae/
 WAE WAE App Rollout

Deployer (1) Install Collector

Sys Config/
Model
Building
App
Development/
Testing

Deployer

• When do I use NSO and when to use

OSC?
• Use OSC for PCEP and BGP-LS
• Use NSO for existing hardware

• OSC is included in the WAE distribution

• NSO is a separate installation
•
WAE REST
Need to configure WAE to tell it which Network
Planning
Failure
Analysis Weather Map Trending
APIs

deployer to use WAE Design (Planning) WAE Live (Analytics) Other Apps

WAN Automation Engine

Analytics Optimization and Prediction Calendaring

Current Model Network Modeler New Model

Network
Collector Deployer
Interface

SNMP CLI NetFlow BGP-LS NMS/EMS NETCONF/YANG EMS/NMS PCEP

WAE WAE App Rollout

Deployer (2) Install Collector

Sys Config/
Model
Building
App
Development/
Testing

Deployer

Three alternate deployer options outside of NSO and OSC

1. Set the WAE deployer to ‘template’. This will create a text file for another
deployer system to use
2. Use a 3rd party ODL.
• The ODL interfaces must be the same version we use (Helium SR1). This is
possible, but not supported or tested
• You need the WAE plug-in. Put this file in the ODL ‘deploy’ directory.

3. WAE is OSGI based, you can write southbound plug-ins in java and load
them in the WAE container
 WAE WAE App Rollout

App Rollout Install Collector

Sys Config/
Model
Building
App
Development/
Testing

Deployer

• End-game is a solution that solves the problem

• Success Factors that we have observed with many network operators
• Know your “pain points”
• Stakeholder planning is key – including vendor dependencies
• Develop phased approach to testing/deployment
• Training in software is a must
• Option to start with offline planning  online deployment
• Data collection a prerequisite – from devices and/or existing stats
warehouse(s)
• Learn about REST APIs
• We are here to help!!
Summary
Summary (1)
• WAN is that critical resource in network and service deployments
• TE disciplines  mature offline and now online tools deployed
• SDN WAN Orchestration involves an advanced suite of:
• analytical, optimization/planning and calendaring applications
• accessible via APIs to operators, customers and 3rd parties
• Independent of southbound collection and programming methods

• Real focus is on the applications!

SDN WAN Orchestration: Optimize, Add More Traffic, Enable

New Services
Summary (2): How is WAE Positioned?
• WAE is an application engine that
• enables applications to make queries about demands
• run simulations
• request demand placement on the network

• WAE uses topology and traffic abstraction

• by collecting information from the network
• in building the model

• WAE is NOT a controller !!!!!!

• But NSO and Open SDN Controller can compliment WAE
Our New Field of Play …
Software
Engineering

Cloud
Where NG
Operators Need
Tail-F NCS
Play

Stateful PCE
MPLS TE

ISIS
Segment
Routing
BGP Network
Engineering
 Reference

Getting Started with WAE

• dCloud – fully functional demos
http://dcloud.cisco.com/
• Search for “WAE” or” MATE”
• WAN Automation Engine with virtual network and
Demo Applications
• WAE Design Application
• WAE Analytics Application
• WAE Collection Module

• WAE home page on cisco.com:

http://www.cisco.com/go/wae
 Reference

Getting Started with the WAE API

• DevNet – WAE Tutorials,
POSTMAN libraries and API
Documentation
https://developer.cisco.com/site/wae/index.gsp

• Learning Labs – Hands on with

the WAE APIs over in the
DevNet zone
https://learninglabs.cisco.com/#/home
Dare to Develop Reference
Discover WAE and Coding examples at DevNet
Complete Your Online Session Evaluation
• Give us your feedback to be
entered into a Daily Survey
Drawing. A daily winner
will receive a $750 Amazon
gift card.
• Complete your session surveys
though the Cisco Live mobile
app or your computer on
Cisco Live Connect.
Don’t forget: Cisco Live sessions will be available
for viewing on-demand after the event at
CiscoLive.com/Online
 Reference

Continue Your Education

• Demos in the Cisco campus
• Devnet
• Walk-in Self-Paced Labs
• Table Topics
• Meet the Engineer 1:1 meetings
• Related sessions
Thank you
References
References 1
• North-Bound Distribution of Link-State and TE Information using BGP
• https://datatracker.ietf.org/doc/draft-ietf-idr-ls-distribution/
• RFC 4655 A Path Computation Element (PCE)-Based Architecture
• https://datatracker.ietf.org/doc/rfc4655/
• RFC 5440 Path Computation Element (PCE) Communication Protocol (PCEP)
• https://datatracker.ietf.org/doc/rfc5440/
• Applicability of Stateful Path Computation Element (PCE)
• https://datatracker.ietf.org/doc/draft-ietf-pce-stateful-pce-app/
• PCEP Extensions for PCE-initiated LSP Setup in a Stateful PCE Model
• https://datatracker.ietf.org/doc/draft-crabbe-pce-pce-initiated-lsp/
• ReST
• http://www.ics.uci.edu/~fielding/pubs/dissertation/rest_arch_style.htm

• OpenFlow on ASR9K
• https://supportforums.cisco.com/sites/default/files/asr9k-sdn-openflow-whitepaper.pdf
References 2
• Implementing MPLS Traffic Engineering (incl. Stateful PCE)
• http://www.cisco.com/c/en/us/td/docs/routers/asr9000/software/asr9k_r5-
1/mpls/configuration/guide/b_mpls_cg51xasr9k/b_mpls_cg51xasr9k_chapter_0101.html
• Implementing BGP (incl. BGP LS)
• http://www.cisco.com/c/en/us/td/docs/routers/asr9000/software/asr9k_r5-
1/routing/configuration/guide/b_routing_cg51xasr9k/b_routing_cg51xasr9k_chapter_010.html

• Netconf/Yang Tutorial
• http://www.ietf.org/edu/documents/2012-ietf-84-netconf-yang.pdf

• SDN Tutorials
• http://sdntutorials.com/opendaylight-netconf-restconf-and-yang/

• Netconf Central
• http://www.netconfcentral.org/

• YANG Central
• http://www.yang-central.org/twiki/bin/view/Main/YangTools
References 3
• https://tools.ietf.org/html/rfc6571
• draft-francois-segment-routing-ti-lfa
• http://www.segment-routing.net/
• https://wiki.onosproject.org/display/ONOS/Segment+Routing
XR Configuration and Commands for PCEP MPLS
Router Configuration Useful commands
ipv4 unnumbered mpls traffic-eng Loopback0 RP/0/0/CPU0:alb#show mpls traffic-eng pce peer
Mon May 11 02:30:06.558 UTC
router isis pce Address Precedence State Learned From
address-family ipv4 unicast --------------- ------------ ------------ --------------------
mpls traffic-eng level-2-only 172.29.104.110 255 Up Static config
mpls traffic-eng router-id Loopback0 #
… RP/0/0/CPU0:alb#mpls traffic-eng pce activate-pcep all
mpls traffic-eng Mon May 11 02:30:44.585 UTC
… ################## configuration mode
pce interface tunnel-te1234
peer ipv4 198.18.1.26 destination 28.28.28.28
! path-option 1 dynamic
stateful-client pce
instantiation delegation
! !
! !
auto-tunnel pcc ##############
tunnel-id min 1 max 99 RP/0/0/CPU0:alb#show mpls traffic-eng tunnels br
! TUNNEL NAME DESTINATION STATUS STATE
reoptimize timers delay installation 0 tunnel-te1234 28.28.28.28 up up
! >tunnel-te1 28.28.28.28 up up
XR Configuration and Commands for PCEP SR
Router Configuration Useful commands
ipv4 unnumbered mpls traffic-eng Loopback0 RP/0/0/CPU0:alb#show mpls traffic-eng pce peer
router isis pce Mon May 11 02:30:06.558 UTC
address-family ipv4 unicast Address Precedence State Learned From
mpls traffic-eng level-2-only --------------- ------------ ------------ --------------------
mpls traffic-eng router-id Loopback0 172.29.104.110 255 Up Static config
segment-routing mpls #
interface Loopback0 ################## configuration mode
address-family ipv4 unicast interface tunnel-te1234
prefix-sid absolute 16142 pce
… delegation
mpls traffic-eng !
… !
pce ##############
peer ipv4 198.18.1.26 RP/0/0/CPU0:alb#show mpls traffic-eng tunnels br
! TUNNEL NAME DESTINATION STATUS STATE
segment-routing tunnel-te1234 28.28.28.28 up up
stateful-client >tunnel-te1 28.28.28.28 up up
! #sh mpls forwarding tunnels
auto-tunnel pcc Tunnel Outgoing Outgoing Next Hop Bytes
tunnel-id min 1 max 99 Name Label Interface Switched
! ------------- ----------- ------------ --------------- ------------
te1234 (SR) 24002 Gi0/0/0/0 172.16.1.1 0
WAE Configuration to use PCEP

Edit the WAE configuration files:

• $WAE_HOME/wae-core/etc/com.cisco.wano.nsps.deployer.pcep.cfg
• Change line 8 to pcepDeployerProxy=odlPcepDeployerProxy

• $WAE_HOME/software/wae-core/etc/com.cisco.wano.nsps.deployer.cfg
• Change line 9 to nonPcepDeployer=pcep
XR Configuration and Commands for BGP-LS
Router Configuration Useful commands
router isis pce-poc RP/0/0/CPU0:alb#show mpls traffic-eng pce peer
is-type level-2-only Mon May 11 02:30:06.558 UTC
distribute bgp-ls level 2 Address Precedence State Learned From
… --------------- ------------ ------------ --------------------
router bgp 72 172.29.104.110 255 Up Static config
bgp router-id 30.30.30.30 #
… ################## configuration mode
address-family link-state link-state interface tunnel-te1234
! pce
neighbor 198.18.1.26 delegation
remote-as 72 !
update-source MgmtEth0/0/CPU0/0 !
address-family ipv4 unicast ##############
route-reflector-client RP/0/0/CPU0:alb#show mpls traffic-eng tunnels br
! TUNNEL NAME DESTINATION STATUS STATE
address-family link-state link-state tunnel-te1234 28.28.28.28 up up
route-reflector-client >tunnel-te1 28.28.28.28 up up
! #sh mpls forwarding tunnels
! Tunnel Outgoing Outgoing Next Hop Bytes
Name Label Interface Switched
------------- ----------- ------------ --------------- ------------
te1234 (SR) 24002 Gi0/0/0/0 172.16.1.1 0
WAE Configuration for BGP-LS
• Edit the CDL configuration files:
/opt/cariden/software/wae-cdl/etc/opendaylight/karaf/41-bgp-example.xml
• On line 61, a commented section begins. end the comment on line 68
• On line 72, change the <host> to the IP address of your BGP-LS speaking router:
<host>10.11.12.27</host>
• On line 101, set the AS: <local-as>1</local-as>
• On line 102, set the IP address to local WAE server IP which will be used for the BGP session: <bgp-
id>192.168.139.7</bgp-id>
• If using IANA LSA you may need to modify this file: $WAE_HOME/wae-cdl/etc/opendaylight/karaf/31-
bgp.xml
• On line 67 change to: <iana-linkstate-attribute-type>true</iana-linkstate-attribute-type>
Application Categories
Traffic engineering Operational Simplification
Tunnel split merge Coordinated Maintenance
Tunnel balancer MATE Live
Latency based routing Bandwidth Calendaring (time shifting traffic)

Cross Domain Reactive Control

BW Calendaring (with business system integration) WAE reacts to conditions without human
involvement
Intelligent workload placement