Enterprise MPLS:

Customer Case
Studies
George Bekmezian – Solutions Architect CVE Technologies

Robert King - Deputy Director of Telecommunications, L.A.

County

BRKMPL-2110
Cisco Spark
Questions?
Use Cisco Spark to communicate
with the speaker after the session

How
1. Find this session in the Cisco Live Mobile App
2. Click “Join the Discussion”
3. Install Spark or go directly to the space
4. Enter messages/questions in the space

Cisco Spark spaces will be cs.co/ciscolivebot#BRKMPL-2110

available until July 3, 2017.

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Agenda

• Introduction of Session and

Speakers
• MPLS Use Case – LDS Church
• MPLS Use Case – L.A. County
• Open Q & A
Enterprise MPLS:
Customer Case Studies
Speaker #1… George Bekmezian

George (CCIE R&S and Security 10704) is a Solutions Architect for CVE Technologies Group.

In his current role, George provides presales design and engineering support for some of CVE's
largest enterprise customers. He has over 20 years of experience in the information technology
space.

George has been involved in designing, implementing and supporting complex and highly
available networks in both private and DoD markets. Before joining CVE, he worked for Cisco for
over 9 years in pre-sales, post-sales and BU roles. He was the Cisco SE supporting the LDS Church
for two years and worked as a senior engineer for the LDS Church for a year.
Enterprise MPLS:
Customer Case Studies
Speaker #2… Robert King

Robert is the Deputy Director of Network and Collaboration Services for the County of
Los Angeles. Specifically he works for a central IT agency called Internal Services with
the county of LA.

Robert has worked in the Telecom/IT space for 36 years, starting as an

hardware/software engineer, moving into a sales engineer role, and now an executive
manager at County of Los Angeles for the last 16 years.
 MPLS Use Cases –
LDS Church

George Bekmezian – Solutions Architect CVE Technologies

BRKPL-2110
Agenda

• Introduction
• Why MPLS in our Campus?
• Design Overview
• Operational Impact
• What’s Next
• Conclusion
About the LDS Church & its Environment

The LDS Church

• The Church of Jesus Christ of Latter-day Saints (LDS)
• Also known as the Mormons

The LDS Church HQ Environment

• Approximately 9,000 workforce users
• Campus MPLS Network covers 33 sits in the Salt Lake City MAN
• 4 main datacenters: 3 in Utah, 1 in Virginia

BRKMPL-2110 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
Why MPLS was chosen at the LDS Church?

BRKMPL-2110 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
Why MPLS was chosen at the LDS Church?
Meltdown was imminent

BRKMPL-2110 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
Ten Years Ago
Building A Building B
• Layer 2 sprawl
• Spanning tree loops
• Service interruptions

Routers

Switches

Layer 2

Layer 3

BRKMPL-2110 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
This wasn’t working…

BRKMPL-2110 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
This wasn’t working…
Jobs were hanging on the brink…

BRKMPL-2110 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
Requirements
• Stability
• Stability
• Stability
• Segmentation
• Supportability

BRKMPL-2110 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
Options Considered #1
Building A Building B
Traditional Layer 3 Access

• Distributed management
• Distributed enforcement
• Decentralized, complex

BRKMPL-2110 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
Access Control List (ACL) Limitation
ACL enforcement is intended for broad definition of access policy. Firewalls are
typically required to provide more granular policy control further upstream in the
network or closer .to the protected hosts.
Hardware limits often restrict the number of ACL rules on the access device to
a few entries, especially on older switches (TCAM memory limits).
Device-level ACL management is cumbersome, even with centrally managed
“downloadable ACLs” (dACLs) – especially if policies change frequently

BRKMPL-2110 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
What is “Network” Virtualization?
One physical network supports multiple virtual networks

Virtual Network Virtual Network Virtual Network

Actual Physical Network

BRKMPL-2110 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
Traditional Segmentation
• Fundamentally VLAN based
VLAN VLAN
• Every segment is a separate
Enforcement 10 20
VLAN / Subnet /
IP based policies.
• Segment to segment ACLs, Firewall
rules VRF-10 VRF-20
communication governed by IP
routes and IP based policies
• Classify assets in to VLAN, Propagation Campus LAN
transport context in L2 (VLAN Carry segment
tag) / L3 (IP address / VRF), context over
the network Subnet Subnet
Enforce based on IP-ACLs through VLAN 10.10.X.X 10.20.X.X
tags / IP
• Unless VLANs are completely address / VRF
isolated through the use of Virtual
Routing and Forwarding (VRFs), Classify
ACLs or other firewall services Static /
are needed at the VLAN Dynamic VLAN
boundaries. assignments VLAN-10 VLAN-20

BRKMPL-2110 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
 Segmentation
Segmentation:
• Define business critical/relevant zones

Employees Production

Development

Micro Segmentation:
• Define segmentation policy within zones
• Ex: user to user policy

BRKMPL-2110 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
Routing Protocols in MPLS VPN

BRKMPL-2110 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
VRF-Lite End-to-End
How Does It Work?
Create L2 VLANs at the edge of the network
and trunk them to the first L3 device –or L3 access VLAN 10
VLAN 20

VRFs need to be defined on each L3 device,

Map the VLANs to a VRF VLAN 11 VLAN 12
VLAN 21 VLAN 22
IGPs are configured for each VRF on each
L3 device
IGPs
Trunks need to be configured to carry each
of the VRFs VLAN 15
Create sub-interfaces and map them to the correct VLAN 13 VLAN 25
VLAN 23
VRF
VLAN 14
Traffic is now carried end-to-end across the network VLAN 24
VLAN 16
maintaining logical isolation between the defined VLAN 26
groups

BRKMPL-2110 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
Enterprise Network Virtualization with VRF-Lite
Device virtualization Data path virtualization
Hop-by-Hop

VRF
VRF 802.1q
Global

Multi-Hop
IP
Virtual Routing & Forwarding Table

Every Hop Must be VRF Aware

BRKMPL-2110 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
Options Considered #2
VRF-Lite Building A Building B

• Distributed management
• Static
• Centralized enforcement
• Complex – doesn’t scale

BRKMPL-2110 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
MPLS Data Plane

BRKMPL-2110 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
Options Considered #3
MPLS Site A Site B

• Hybrid management
• Dynamic
• Centralized enforcement
• Simple (Relatively)

P Router

PE Router

L2 Access

BRKMPL-2110 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
MPLS Segmentation

BRKMPL-2110 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
Routing Protocols (Route Reflectors)
MPLS
Site A Site B
• OSPF (IGP)
• MP-BGP
• Route Reflectors

Route Reflector

P Router

PE Router

L2 Access

BRKMPL-2110 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
Current Campus Access Design
MPLS Site A Site B

• Subinterfaces on PEs
• No spanning tree

P Router

PE Router

L2 Access

BRKMPL-2110 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
Current Campus Access Design

BRKMPL-2110 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
 BLDG
Transit Design (DCs) Core
LDP
MPLS
FW DC
• ASA Cluster Advertises Core Core
defaults into VRFs Fabric
EIGRP
Path
• FW PEs advertise
summaries to ASAs ASA
Cluster
• All security policies are
centralized in the ASAs P Router

PE Router

Fabric Path

UCS FIs

BRKMPL-2110 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
 BLDG
Transit Design (DCs) Core
LDP Campus Client
MPLS Network (Data VRF)

FW DC
• ASA Cluster Advertises Core Core
defaults into VRFs Fabric
EIGRP
Path
• FW PEs advertise
summaries to ASAs ASA
Cluster
• All security policies are
centralized in the ASAs P Router

PE Router

Fabric Path

Server
(Application VRF) UCS FIs

BRKMPL-2110 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
 LDS Recipe for Success
• Avoid layer 2 extensions (single instance in
prod.)
• Pair of out of band Route Reflectors Per Site
• No advanced MPLS features (traffic
engineering,
• Enterprise common services zone bypasses
firewalls from user zones (DNS, DHCP)
• Datacenter common services zone bypasses
firewalls for datacenter zones (backup,
loadbalancers, storage)
• Avoid VRF Sprawl

BRKMPL-2110 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
LDS Recipe for Success
Enterprise common services zone bypasses firewalls

BRKMPL-2110 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 34
LDS Recipe for Success
Enterprise common services zone bypasses firewalls

BRKMPL-2110 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
LDS Recipe for Success
Enterprise common services zone bypasses firewalls

BRKMPL-2110 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
Operational Impact of Enterprise MPLS
• Perception

BRKMPL-2110 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 37
Operational Impact of Enterprise MPLS
• Perception
• Reality

BRKMPL-2110 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 38
Operational Impact of Enterprise MPLS
• Perception
P Router
• Reality
PE Router
• Multicast – Emergency notification services

L2 Access
MDT

Multicast
Receivers

Multicast Multicast
Receivers Source
BRKMPL-2110 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 39
Operational Impact of Enterprise MPLS
• Perception
• Reality
• Multicast
• Tools
• Config Management
• NetFlow

BRKMPL-2110 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 40
Operational Impact of Enterprise MPLS
• Perception
• Reality
• Multicast
• Tools
• Security
• Inline taps
• VRF Aware SIEM

BRKMPL-2110 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
Looking Ahead
• Revisit segmentation options
• TrustSec / SGTs?
• Campus Fabric?
• Stay the course?

BRKMPL-2110 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
MPLS Use Case –
County of Los Angeles
Robert King
Deputy Director of Telecommunications, L.A .County

BRKMPL-2110
Agenda

• Overview – County of Los

Angeles
• Major IT Initiatives
• Current MPLS Infrastructure
• Why Private MPLS Network?
• Benefits for the County
• MPLS Roadmap
• Considerations
Overview - County of Los Angeles

• 108,000 employees
• 10 million residents
• 4,000 square miles
• $30B recommended budget for fiscal year 17-18
• Largest municipal government in the United States
• Only four States have larger budget than County of Los Angeles
• Governed by five elected Supervisors
• Four of five are female
• Supervisor-appointed CEO also female
• 39 departments and agencies
• Central IT agency via Internal Services Department (ISD)

BRKMPL-2110 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 45
Major IT Initiatives
• New Data Center
• Signed 10 year lease with T5 Los Angeles; ribbon cutting held May 31
• Purchased some new equipment and migrating existing infrastructure
• Complete by Dec. 31, 2017
• Strategic goal to centralize all existing 60+ “data centers” by 2020
• Start department (customer) workload migration in Jan. 2018

• New WAN infrastructure

• Support new data center
• Leveraging existing MPLS architecture
• Sustain and refresh existing IT infrastructure
• Security infrastructure + ATA services
• LAN, WAN, WiFi, Extranet
• Unified Communications & Contact Centers (VoIP)
• Videoconferencing infrastructure
• Cisco Meeting Server; room systems; Jabber; Skype4Business
• 24/7/365 NOC

BRKMPL-2110 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 46
Current MPLS Infrastructure
• Serving 39 Departments and Agencies
• e.g. Animal Care and Control, Assessor, Auditor-Controller, Beaches and
Harbor, Board of Supervisor, Chief Executive Office, Children and Family
Services, District Attorney, Fire, Health, Probation, Public Library, Public
Social Services, Public Works, Registrar-Recorder/County Clerk, Treasurer
and Tax Collector…
• Enterprise Network (WAN) span to 960+ Locations
• Dual hub and spoke architecture
• Leveraging multiple local carriers
• AT&T, Verizon, Frontier, Spectrum (Time Warner) Metro Ethernet
• Handful of private fiber

BRKMPL-2110 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 47
Enterprise Network Diagram
Los Angeles County Enterprise Network
Datacenter 2
Datacenter 1 Internet

Datacenter Interconnect
LA County
MPLS WAN
900+ Sites

Hub Site 1 Hub Site 2

Dist 1 Dist 2 Dist 3 Dist 4 Dist 1 Dist 2 Dist 3 Dist 4

Network Network
Analyzer Analyzer

Firewall Firewall
Fusion Fusion
Edge Edge

Core Core

Route Route
Edge Edge
Reflector Reflector

Extranet Extranet

Extranet
Cerner, State,
Sheriff, etc.

BRKMPL-2110 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 48
Why Private MPLS Network?
• Why did we implement our private MPLS?
• Restrain rampant and widespread network worm and virus after Y2K
• Separate and isolate departments for security concerns
• Limit virus spread between County departments
• Some departments didn’t implement best practices
• Need for virtual private network to meet certain audit requirements
• Meet certain unique departmental needs and “demands”

• Why didn’t we buy MPLS from Carriers?

• It was not generally available at that time (2000/2001)
• Carrier-based MPLS is not as flexible or adaptable
• Lack of control; delay in changes
• Cost considerations as ISD already has network engineering resources

BRKMPL-2110 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
Benefits for the County
• Inherent private MPLS benefits:
• More operational control
• No limit on number of VPNs
• Supports multicast
• Video streaming Board of Supervisor meetings
• Retain internal technical expertise
• Direct control of infrastructure
• Rapid deployment and changes
• Control of operating costs
• Simplify departmental budgeting
• Transparent network design
• Better accountability of sensitive and protected data (e.g. PHI,
HIPAA)
BRKMPL-2110 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 50
MPLS Roadmap
• Complete MPLS migration for 100% of remote site
• 20 Frame Relay sites remain
• Plan for 1G WAN bandwidth as baseline “norm”
• Plan for 40G-100G backbone
• Plan to add OSPF protocol to WAN backbone
• Augment existing EIGRP protocol
• Enable Traffic Engineering capability in 2018

• Collaborate with Cisco on future Network Automation Capabilities

• Future MPLS Core Integration into ACI and Tetration

BRKMPL-2110 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 51
Considerations
Total cost of ownership (TCO): Build vs. Buy
• Build a private MPLS
• Capital cost to procure equipment
• Operating costs - annual hardware/software/license refresh
• Need for highly skilled internal technical resources and training

• Buy MPLS
• Establish contract(s) with carriers or network providers
• Clearly defined SLAs
• Customer portal for network changes and dashboard visibility
• Transition planning for onboarding

BRKMPL-2110 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 52
Considerations

County of Los Angeles network team:

• Seven network design engineers and one senior network architect
• Five network operations engineers
• Team of 25 field network technicians
• NOC personnel
• 14 positions covering three shifts

BRKMPL-2110 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 53
Q&A

BRKMPL-2110 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 54
 Complete Your Online
Session Evaluation
• Give us your feedback to be
entered into a Daily Survey
Drawing. A daily winner will
receive a $750 gift card.
• Complete your session surveys
through the Cisco Live mobile
app or on www.CiscoLive.com/us.

Don’t forget: Cisco Live sessions will be

available for viewing on demand after the
event at www.CiscoLive.com/Online.

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Continue Your Education
• Demos in the Cisco campus
• Walk-in Self-Paced Labs
• Lunch & Learn
• Meet the Engineer 1:1 meetings
• Related sessions

BRKMPL-2110 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 56
Thank you