#CiscoLive

Redefining Network
Assurance
Challenging the Hybrid World with ThousandEyes

Manny Garcia, Principal Engineer, Cisco IT

@supermanny81
BRKAPP-2545

#CiscoLive
Cisco Webex App

Questions?
Use Cisco Webex App to chat
with the speaker after the session

How
1 Find this session in the Cisco Live Mobile App

2 Click “Join the Discussion”

3 Install the Webex App or go directly to the Webex space Enter your personal notes here

4 Enter messages/questions in the Webex space

Webex spaces will be moderated

by the speaker until June 9, 2023. https://ciscolive.ciscoevents.com/ciscolivebot/# BRKAPP-2545

#CiscoLive BRKAPP-2545 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
 • About Cisco IT and Me
• Network Assurance
Innocence
• ITSM to the Rescue?

Agenda • Impacts to Proactive and

Reactive Monitoring
• Our Next Steps and Recap

BRKAPP-2545 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
Cisco IT and
Me…
 Cisco at a Glance
Employee Distribution Global Cisco Distribution
Corporate Engineering APJC
Functions 17%

36%
2,483 7,194 27%

81,000 Routers LAN Switches 95

Employees Countries

22%
135,000 370 54%
Connected
Stakeholders Offices
19%

Customer
25%
Sales &
12,553
Experience Marketing Unified Computing Billion DNS EMEAR Americas
System Servers requests per day

22,438 59,686 57,875 PB 1.5 M 6.25M

TelePresence Virtual Machines Mobile Devices Overall Usable Webex Meetings DNS Threats Requests
Units Storage per Month Blocked per day
Data as of June 2022 – Cisco Public #CiscoLive BRKAPP-2545 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
Major Cisco Locations
Data Centers, engineering centers, and sales near customers

Data as of June 2022 – Cisco Public #CiscoLive BRKAPP-2545 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
Our Strategic Principles
Guiding a new way of working

Buy where available, build only for competitive advantage

Drive for self-service,

deliver a fully automated experience (NoOps)

Deliver a Multi-Cloud strategy for Cisco (Private/Public/Hybrid)

#CiscoLive BRKAPP-2545 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
Full Stack Observability in IT - Conceptual
“Hourglass Design”
(multiple sources
feed into one
reconciled model,
repeated)

Normalize and
Standardize Events,
Alerts and
Inventory

Support for multiple

monitoring systems

#CiscoLive BRKAPP-2545 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
Cisco IT Adoption
Key Features
• HTTP Synthetic Transactions
• Network Visualization
9,350 Tests
• DNS Availability and Performance
• User Experience Monitoring with
Endpoint Agents
7,373 1,729 40
• AppD/TE Integration * HTTP Server Agent to Server Web Transaction

• Self-service onboarding

692 730,000 • Automated provisioning with

business logic controls
Applications in IT ThousandEyes k-units • CMDB Integration
• Standardized alerting and
using ThousandEyes used each month exporting to AIOps

#CiscoLive BRKAPP-2545 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
Network
Assurance
Innocence
Historic traffic flows
The age of perimeter-based security, assurance, and almost everything else

Internet

TR AFFIC TR AFFIC

Internal 80% Internal 80%

Internet 20% Internet 20%
Security stack

WAN VPN

Branch offices HQ Roaming/mobile

#CiscoLive BRKAPP-2545 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
 Some of our use cases
Organization Description Test Type Comments Target Quarter

Cloud Agent to Server (TCP Monitoring the availability and latency of Cisco's VPN infrastructure
VPN In Production
443) from external sources.
Inbound Internet Access Cloud Agent to Server (ICMP) Ensure external traffic can reach Cisco's Network. In Production

Network Services Enterprise Agent to Enterprise

CBB (Cloud Backbone) Monitor Cisco's Cloud backbone In Production
Agent/Server (ICMP)
Enterprise Agent to Enterprise
CAPNet Monitor Cisco's backbone network. In Production
Agent (ICMP)
BGP BGP BGP Hijack In Production
Agent to Server(Cloud & These tests results are based on application/VPN based https. ESP
ISP Outage Visibility In Production
Enterprise agents) https Mapping missing today
Agent to Server(Cloud &
Https tests are running successfully, however regarding transaction
SaaS Vendors Availability Enterprise agents) https & In Production
tests there are few limitations regarding MFA with the TE production.
Transaction tests
Helpful when troubleshooting provider issues / understanding large
Enterprise Internet Insight Agent to server(Cloud) https In Production
outages.
Operations Center
EOC has built dashboards used to view VPN, SaaS and Cisco TV
TE Metrics Dashboard view N/A In Production
metrics
Currently Alarms are set such that, ODM's receive the alert and we
VPN Availability Agent to server(Cloud) https In Production
engage teams where necessary.
Endpoint Agent to the server
Endpoint Monitoring To monitor performance issue regarding an outage. In Production
(https)
Baseline and then ongoing measurement of user experience for
Teleworker Experience (NaaS) Desktop agent to Cloud agents In Production
teleworkers in Customer Zero environment using the endpoint agent
- Pre and post CR validation. Did we maintain or improve performance
Customer Zero, Test units for:
of network traffic?
Network Services Enterprise agent
Secure Branch (NaaS) - Proactive alerting for XaaS degradation eg Webex or O365 In Production
Cloud agent
- Troubleshooting network issues across LAN WAN and Internet
End point agents
(Enterprise + End point)

#CiscoLive BRKAPP-2545 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
Health of CAPNet*
Cisco All Packet Network – our backbone

#CiscoLive BRKAPP-2545 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
CAPNet Capacity
200+ monitored devices

#CiscoLive BRKAPP-2545 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
SaaS Availability

#CiscoLive BRKAPP-2545 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
VPN Overview

#CiscoLive BRKAPP-2545 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
Alerting Through
Thousand Eyes

• How: Enable a “Edit Custom

Webhook Integration” **
• Why: Enable more reactive
responses to outages/user network
performance issues

BRKAPP-2545 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
What did it take?

176 AGENTS 210 TESTS 200 DEVICES

#CiscoLive BRKAPP-2545 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
A whole new topology…
People moved and applications changed. Cardinality ROSE!

EHR Platform

CLOUD Artificial
Intelligence
PROVIDERS
SERVICE
PROVIDERS

COLOCATION
SECURITY

SaaS
Data

Medical
Imaging

HOSPITAL CLINIC DATA CENTER EDGE | IOT & MDI

#CiscoLive BRKAPP-2545 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
Changes in the types of traffic and destinations
Now everything is upside down
Internet
SaaS IaaS
Private cloud Browsing

TR AFFIC TR AFFIC

Internal 20% Internal 20%

Internet 80% Internet 80%
Bottle neck

MPLS VPN

Branch offices HQ Roaming/mobile

#CiscoLive BRKAPP-2545 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
The Visibility Gap
What network engineers see

ENTERPRISE WAN VISIBILITY GAP APPLICATION STUFF

Branch
INTERNET
?
Private DC

Branch
(ISPs, DNS, CDNs, SWGs) ?
SaaS

#CiscoLive BRKAPP-2545 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
The Visibility Gap
What literally everyone else sees!

Network
Stuff

#CiscoLive BRKAPP-2545 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
ITSM to the
Rescue
Cisco IT is organized as an “IT as
a Service” provider. We provide
about 100 separate services –
and over 1000 service offerings –
to Cisco.
Application and Service Definitions

#CiscoLive BRKAPP-2545 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
Application Monitoring for NaaS

#CiscoLive BRKAPP-2545 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
 IT Service Management Integration
Overview

People Pipelines
Enterprise Service Platform
Technology: ServiceNow

REST APIs CMDB

TE Integration Controller

Web API
CHAIN
Technology: Elasticsearch, Kibana,
Kafka
Worker Event Queue

#CiscoLive BRKAPP-2545 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
 IT Service Management Integration
Monitoring your application, application module, or service endpoint

People Pipelines
Enterprise Service Platform
Technology: ServiceNow
Click, Click YAML
REST APIs CMDB
1
Outage Alerts
2
5 TE Integration Controller

Web API
CHAIN
Technology: Elasticsearch, Kibana,
Kafka 3
Worker 4 Event Queue

6
#CiscoLive BRKAPP-2545 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
 IT Service Management Integration
Change Management

People Pipelines
1 Enterprise Service Platform
Technology: ServiceNow

CMDB

Outage Alerts
2
6 TE Integration Controller
3
API
CHAIN
Technology: Elasticsearch, Kibana, 4 MID Servers
Kafka
Worker 5 Event Queue

#CiscoLive BRKAPP-2545 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
Integrations double your value
• Applications and Services are monitored, and events are managed
in much greater detail
• We went from using Thousand Eyes in specific use cases in
infrastructure to something that is consumed in all of IT
• Expanding the audience to the owners of a service helped network
and service teams work better together
• We were able build were necessary and buy most of what we
needed

#CiscoLive BRKAPP-2545 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
Impacts on
Proactive and
Reactive
Monitoring
A corpus of tests

• Preconfigured tests can be

made available to all teams
• BYOT, the sky is the limit
• Everyone gets a dashboard!
• One team, shared knowledge
and responsibilities
• It’s part of the machine!
Tied to change and event
management systems

BRKAPP-2545 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
What about endpoint agents?

#CiscoLive BRKAPP-2545 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 34
Reactive Monitoring with Endpoint Agents

#CiscoLive BRKAPP-2545 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
Shared Visibility

#CiscoLive BRKAPP-2545 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
On demand tests of our services

#CiscoLive BRKAPP-2545 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 37
#CiscoLive © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
Pro-active support for Wireless devices

• Measure over 30 days from managed desktop device

• Highlight poor wireless signal quality (65% and below)
• Support team investigates and outreaches to user where required
• Once data set has matured, automate the insights

#CiscoLive BRKAPP-2545 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 39
Next Steps
Next Steps for the Endpoint Agent
> 1% of overall tests
• More tests!
• Passive monitoring
• More insights to our end users
• Deploy endpoint agents to
everyone

Cloud and Enterprise Agent Endpoint Agent

#CiscoLive BRKAPP-2545 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
DNAC + Thousand Eyes

#CiscoLive BRKAPP-2545 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
DNAC + ThousandEyes
Get a view of what tests your running on premise

#CiscoLive BRKAPP-2545 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 43
DNAC + ThousandEyes
CBAR and TE test transactions

#CiscoLive BRKAPP-2545 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 44
Fine

• Shared about Cisco and Cisco IT

• We covered some of our Thousand Eyes use cases for our
backbone, SaaS and VPN
• The topology has changed, it’s pushing us to know more about the
applications and how people get there
• We started to better define our applications, close the gaps of
visibility, and better react to issues
• We’re looking to continue to get more value out of investments,
help change the way we work

#CiscoLive BRKAPP-2545 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 45
Additional Resources
DevNet Resources:
https://developer.cisco.com/codeexchange/github/repo/CiscoIT-DAE/ThousandEyes-
Transactions/
Custom Web Hooks:
https://docs.thousandeyes.com/product-documentation/best-practices/custom-
webhooks-webex
Full Stack Observability:
BRKCOC-2026 – End to end monitoring an application ecosystem in private and public
cloud. By Clement Joseph.

#CiscoLive BRKAPP-2545 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 46
Fill out your session surveys!

Attendees who fill out a minimum of four session

surveys and the overall event survey will get
Cisco Live-branded socks (while supplies last)!

Attendees will also earn 100 points in the

Cisco Challenge for every survey completed.

These points help you get on the leaderboard and increase your chances of winning daily and grand prizes

#CiscoLive BRKAPP-2545 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 47
 • Visit the Cisco Showcase
for related demos

• Book your one-on-one

Meet the Engineer meeting

• Attend the interactive education

with DevNet, Capture the Flag,
Continue and Walk-in Labs

your education • Visit the On-Demand Library

for more sessions at
www.CiscoLive.com/on-demand

BRKAPP-2545 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 48
Thank you

#CiscoLive
Gamify your Cisco Live experience!
Get points for attending this session!

How:
1 Open the Cisco Events App.

2 Click on 'Cisco Live Challenge’ in the side menu.

3 Click on View Your Badges at the top.

4 Click the + at the bottom of the screen and scan the QR code:

#CiscoLive BRKAPP-2545 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 50
#CiscoLive