BRKPRG 1816

#CLUS
Containers for
Network Automation
and Operations -
IOS-XE and NX-OS
Nagendra Kumar Nainar, Principal Engineer, CX
Yogesh Ramdoss, Principal Engineer, CX
BRKPRG-1816
#CLUS
Cisco Webex Teams
Questions?
Use Cisco Webex Teams to chat
with the speaker after the session
How
1 Find this session in the Cisco Live Mobile App
2 Click “Join the Discussion”
3 Install Webex Teams or go directly to the team space
4 Enter messages/questions in the team space
Webex Teams will be moderated cs.co/ciscolivebot#BRKPRG-1816

by the speaker until June 16, 2019.
#CLUS © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Introduction
Motivation
Make you all aware of advanced architecture and capabilities in our
platforms, and how to bring them into your day-to-day tasks
Containers
in Cisco
Sky is
Network the
Devices? limit!!
#CLUS BRKPRG-1816 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
Agenda
• Virtualization and Containers
• Containers in Cisco IOS-XE
• Containers in Cisco NX-OS
• Summary and Take-aways
Reference 
What's In It For Me?
Session Abstract
• Refresh virtualization and container concepts
• Understand how virtualization and containers

relate to Cisco’s platforms
• Gain knowledge of container and application
hosting capabilities in IOS-XE and NX-OS,
and container networking models
• Understand how containers can be
orchestrated in supported platforms
• Know how to bring these capabilities into
day-to-day operations and administration Courtesy: Reader’s Digest
with few use-cases

Virtualization
and
Containers
Virtualization
Once up on a time….
Minicomputers. Server
Time sharing Consolidation
1970
1990
2015
service DISCO from
Compuserve, Stanford
Tymshare Commodity OS
High cost Evolution of Server
1980
2000
computing Client-Server Virtualization
1960
machines - model. VMware Full

Mainframes Application virtualization
Service bureaus Service Xen Open
based consumption providers source
Litton Mellonics,
Mcdonald-douglas
Evolution of Virtualization
Motivation for Virtualization
• Eliminates the need for • Multiple instances of

dedicated server for each application in different
application. Resource resource allows higher uptime.
• Efficient resource usage Resiliency • Efficient load balancing
Optimization
Virtualization
• Ease of deployment and Cost

Operation. Simplicity • Less Physical entities, floor
• Simplifies workload migration,
Optimization space, power utilization.
upgrade etc. • Reduced OPEX
Virtualization and Cloud Computing
Virtualization Cloud Computing

• Decouples software from underlying • On-demand shared computing resource
hardware and infrastructure delivery
Types of Virtualization
Server Virtualization Network Virtualization
Image vWAAS vSwitch
vCPU vMem vNIC vFirewall vRouter
Virtualization Layer
Physical Hardware
Server Virtualization
Virtual Machine 1 Virtual Machine 2
Virtual Machines
• Hypervisor allows multiple OS to Application Application
share the same hardware
Guest OS Guest OS
• Hypervisor can run on bare metal
or on other OpenStack Platform Virtual HW Virtual HW
• Virtual Machines runs its own
dedicated Operating System Hypervisor
Physical Host
• Ex: KVM, Hyper-V, VirtualBox etc. (CPU, Memory, NIC, Disk)
Curious to know if your Linux

host supports KVM?
KVM Installation
Checking KVM supportability
root@server:-$ kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used
root@server:-$
Updating KVM packages

root@server:-$ sudo apt-get install qemu-kvm libvirt-bin virtinst bridge-utils
Reading package lists... Done
Building dependency tree
Reading state information... Done
bridge-utils is ready the newest version (1.5-9ubuntu1).
bridge-utils set to manually installed.
Server Virtualization
Containers Container 1 Container 2
• Containers are light-weight

compared to Virtual Machine Application Application
• Containers shares the host OS and Binary/Lib Binary/Lib

no dedicated guest OS required
Container Manager
• Package of binaries and libraries
required for the application Host Operating System
Physical Host
• Ex: Linux Containers (LXC), Docker (CPU, Memory, NIC, Disk)
Containers
Container Hosting
BusyBox
Usually 1 application per container - Microservices

LXC Containers
root@server:-$
root@server:-$ egrep ‘(svm|vmx)’ /proc/cpuinfo
Verify if the Processor supports Virtualization root@server:-$
root@server:-$
root@server:-$ sudo apt install lxd
Install Linux container Manager Reading package lists... Done
Building dependency tree
root@server:-$ sudo lxc launch ubuntu:16:04 robot

Creating robot
Launch a container Starting robot
root@server:-$
root@server:-$ sudo lxc list

+---------+----------+--------+--------+--------------+-------------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
List all containers +---------+----------+--------+--------+--------------+-------------+
| first | RUNNING | | | PRESISTENT | o |
+---------+----------+--------+--------+--------------+-------------+
| robot | RUNNING | | | PRESISTENT | o |
+---------+----------+--------+--------+--------------+-------------+
root@server:-$
Docker Containers
Architecture
• Docker Client
• CLI to interface with
Docker
• Dockerfile
• Instruction file to build
Docker Image
• Registry
• Docker image registry
Docker Containers
Dockerfile
Build docker image: docker build FROM python:alpine3.7
COPY . /app1
List the local images: docker images WORKDIR /dir
Start a docker container: docker create <image> RUN pip install -r requirements.txt
List the containers: docker ps CMD [ "python3", "./app1-server_udp.py" ]
root@server:~/python/sbfd (kubernetesproject1-197120)$ docker images

REPOSITORY TAG IMAGE ID CREATED SIZE
<none> <none> 01c4f2d94d18 4 seconds ago 88.9MB
Python alpine 00be2573e9f7 43 hours ago 81.3MB
root@server:~/python/sbfd (kubernetesproject1-197120)$
root@server:~/python/sbfd (kubernetesproject1-197120)$ docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
0dbc6f7cff20 ubuntu "/bin/bash" 12 seconds ago Up 12 seconds practical_keller
root@server:~/python/sbfd (kubernetesproject1-197120)$
Orchestration and Automation
How can I
automate
container
instantiation?
Kubernetes
• Kubernetes is an open source platform for automating,
deploying, scaling, and operating containers
• Kubernetes was created by Google and donated to
Cloud Native Compute Foundation (open source)
• Since kubernetes is open source there are dozens of
projects to enable networking, storage etc.
• Allowing customers to focus on applications instead of
infrastructure
Additional components
• Fluentd – Cluster level logging
Kubernetes Architecture • CoreDNS
• Dashboard
• Kube-dns-autoscaler
Kube-controller-manager handles Cloud-Controller-manager • Heapster – container
nodes, replication, Endpoint handles, nodes, services like Loadbalancers and performance & monitoring
controller joins Services to Pods routes • Event-exporter(fluentd)
Kubernetes Masters Cloud Connector

All Communications
goes through
apiserver
Including UI/CLI/API kubelet
Node agent-
runs Pods,
health checks
Distributed
Key-Value store
kube-proxy
Proxies
TCP/UDP
Kube-scheduler Watches
Binds Pod to Node Kubernetes Workers
Services and
configures
IPTables
Kubernetes Eco System
OpenComponent
Grafana
Visualization APIs Database, Analytics
Middleware App Runtime
SAML
Monitoring OS, Containers DevOps

Security #CLUS BRKPRG-1816 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
Why Containers in Network Devices?
Motivation: Containers on Network Devices
• Network OS Independence
• Limited Kernel dependency
• System Modularity
• Leverage existing toolsets
• Ability to bring application close to the data
• Isolated Application Eco-system
• Application focused deployment
Cisco Software Architecture
Control Plane
(IOS-XE, IOS-XR, NX-OS)
System Infrastructure
Linux Kernel
Platform-Specific Data Plane
Cisco OS Architecture
Between DEMAND and CAPABILITY… there is a bridge.

Containers in
Cisco IOS-XE
Containers in Cisco IOS-XE
• IOS-XE Architecture
• IOS-XE Guest Shell Containers
• IOS-XE IOx App Hosting
• IOS-XE Container Network Model
IOS-XE Architecture
• Linux Kernel based architecture
• IOSd runs as a daemon on Linux Kernel
• 3rd Party Application hosting

capability
• libvirtd is a virtualization management toolkit
• Supports libvirt-lxc, KVM
• Supports Docker image
• Dedicated Filesystem for each

hosted containers
• Logical network connectivity
between the host and the guest
Virtualization Manager
• Virtualization Infrastructure component
3rd 3rd
• Ability to install 3rd party applications Party

APP
Party
APP
Virtualization
• Used in some of the legacy platforms Manager
• CAF augments the infrastructure libvirt
• “virtual-service” CLI based package install Host Kernel
csr1000v-2# virtual-service ?
clear Clear command
connect Connect to an appliance
install Install an appliance package
move Copy files for an appliance package
uninstall Uninstall an appliance package
upgrade Upgrade an appliance package
Cisco Application Hosting Framework (CAF)
Fog
IOx Client
Director
REST
REST
CLI
Cisco Application Hosting Framework
libvirt
Host Kernel
Cisco Application Hosting Framework (CAF)
• New Framework created for application lifecycle management
• Initially developed to host IoT applications
• Expanded the scope of applicability
• Unified Interface for application hosting

• Abstract the underlying resource complexity
• Supported in many IOS-XE Platforms
• Resource availability varies depending on the platform
• ioxclient is used to build native IOx application from docker images
• From 16.12 release onwards, docker image can be natively hosted in
Catalyst 9000 platforms
Cisco Application Hosting Framework
• Flexible Secure Runtime
Environment
• Operating System level Construct
• Virtualization Toolkit
• Structured Open Network Interfaces

• Programmable Interfaces
• Network Datapath for hosted application
• Rich Deployment Service

• SDK Tool kit to build or convert native
apps
Container Hosting
• Guest Shell
• Owner: IOx Guest
3rd
Party
• Type: LXC IOx
shell
App
IOSd VMAN
CAF
• IOx APP Hosting LibVirt
• Owner: IOx
• Type: Custom
Linux Kernel
• VMAN App Hosting

Platform Specific Data Plane
• Owner: VMAN-Agent
• Type: Custom
Platform Comparison
ISR4321 13000 4GB+ Up to 200 GB VM, LXC 2.7
ASR1K 73000 4GB+ Up to 400 GB VM, LXC 2.7
Catalyst 9K 7400 4GB+ Up to 960GB VM, LXC, Docker* 2.7
* from 16.12 release onwards
Feature Matrix
Guest Shell
• Maintain IOS-XE system integrity
• Isolated User Space
• Fault Isolation
• Resource Isolation Linux
applications
• On-box rapid prototyping
• Device-level API Integration
• Scripting (Python)
Guest Shell
• Linux Commands
• Application Hosting
API
• Integrate into your Linux workflow
Network OS
• Integrated with IOS-XE
Enabling Guest Shell
Enable IOX Service
Task 1 - Enable IOX Service Guest Shell N/W Config
C9300# conf t Enable Guest Shell

Enter configuration commands, one per line. End with CNTL/Z.
C9300(config)# iox Optional NAT
C9300(config)# end
C9300#
C9300#
C9300# show iox-service
IOx Infrastructure Summary:
• “iox” command enables the
--------------------------- service on IOS-XE.
IOx service (CAF) : Running
IOx service (HA) : Running
IOx service (IOxman) : Running
• The show command lists the
Libvirtd
C9300#
: Running summary of services enabled.
Enable IOX Service
Task 2 – Guest Shell Network Config Guest Shell N/W Config
C9300# conf t
C9300(config)# interface VirtualPortGroup 0 Enable Guest Shell
C9300(config-if)# ip address 10.1.1.1 255.255.255.0
C9300(config-if)# no shut Optional NAT
C9300(config-if)# end
C9300#
C9300#conf t
C9300(config)# app-hosting appid guestshell
C9300(config-app-hosting)# app-vnic gateway0 virtualportgroup 0 guest-interface 0 Container
C9300(config-app-hosting-gateway0)# guest-ipaddress 10.1.1.2 netmask 255.255.255.0
vEth
C9300(config-app-hosting-gateway0)# app-default-gateway 10.1.1.1 guest-interface 0
C9300(config-app-hosting)# end 10.1.1.2/24
C9300#
• VirtualPortGroup interface acts as gateway for Guest 10.1.1.1/24
Shell Virtual
PortGroup
• Must have a static IP address configured IOS-XE
Enable IOX Service
Task 3 – Enable Guest Shell Guest Shell N/W Config
C9300# guestshell enable Enable Guest Shell

Interface will be selected if configured in app-hosting
Please wait for completion Optional NAT
guestshell installed successfully
Current state is: DEPLOYED
guestshell activated successfully
“guestshell enable”
Current state is: ACTIVATED
guestshell started successfully •
Current state is: RUNNING
Guestshell enabled successfully command will start the
C9300# container
C9300# show app-hosting list • “guestshell destroy” is
App id State
----------------------------------------- used to kill the container
guestshell RUNNING
C9300#
C9300# show app-hosting detail
Guest Shell Verification

App id : guestshell
Owner : iox
State : RUNNING
Application
Type : lxc
Name : GuestShell
Version : 2.4.1(0.1)
Description : Cisco Systems Guest Shell XE for x86
Activated profile name : custom
Resource reservation
Memory : 256 MB
Disk : 1 MB
CPU : 800 units
Attached devices
Type Name Alias
---------------------------------------------
serial/shell iox_console_shell serial0
serial/aux iox_console_aux serial1
serial/syslog iox_syslog serial2
serial/trace iox_trace serial3
Network interfaces
---------------------------------------
eth0:
MAC address : 52:54:dd:af:64:9f
IPv4 address : 10.1.1.2
Port forwarding
Table-entry Service Source-port Destination-port
---------------------------------------------------
C9300#
• IOS XE Guest Shell Containers
• IOS XE Container Network Model
IOx App Hosting
Steps Involved
1 2 3
Get the App Install the Build and
Dev tools/libraries prepare VM
Environment File system
4 5 6
Build IOx Copy the Install and
package package to device host the App
1 2 3
IOx App Hosting Get the App
Environment
Install the
tools/libraries
Build and
prepare VM
Build the App VM File system
• Get any base image for the application
• Build the application sudo yum install <>

systemctl enable serial-getty@ttyS0.service
• Install the binaries and libraries systemctl start getty@ttyS0.service
systemctl status getty@ttyS0.service
• Build and prepare VM File system qemu-img create –f vmdk App1
3
IOx App Hosting Build and
prepare VM
VMDK to QCOW2 Conversion File system
• App VM can be a VMDK file

• Depends on the client used to build
• VMDK to QCOW2 Conversion
• Use qemu-img utility to convert the format
qemu-img convert –O qcow2 Appl.vmdk App1.qcow2
IOx App Hosting
IOx Client Installation
4
package.yaml Build IOx
IOx App Hosting package

descriptor-schema-version: "2.4"
info:
Build IOx Package name: CiscoLiveApp
description: "App Hosting Demo for Ciscolive”
• Create a folder and move version: "1.0"

author-link: "http://www.cisco.com"
the qcow2 file author-name: "Cisco Systems"
app:
• Create package.yaml with type: vm

cpuarch: x86_64
the content shown here resources:

profile: custom
cpu: 200
• Run IOxclient and build memory: 64
disk: 2
the package network:
-
bash$ ioxclient package --name App1 <output-dir> interface-name: eth0
# Specify runtime and startup
startup:
disks:
-
target-dev: "hdc"
file: ”Appl.qcow2"
qemu-guest-agent: True
5
IOx App Hosting

Copy the package
to device
Copy Packages
• Copy the package to USB Flash and
insert on the platform
• External USB must be used to host
the application on Cat9k Platforms
C9300# dir usbflash0:App1xe.tar
Directory of usbflash0:/App1xe.tar
115 -rwx 1418270208 May 3 2019 15:12:32 +00:00 App1xe.tar
31963676672 bytes total (26679033856 bytes free)

C9300#
IOx App Hosting
Configure App-Hosting
• Configure VirtualPortGroup Interface
• Configure DHCP on the hosting Platform
• App-Hosting cannot have static IP address
!
interface VirtualPortGroup1
ip address 10.2.2.1 255.255.255.0
!
ip dhcp pool ciscolive
network 10.2.2.0 255.255.255.0
default-router 10.2.2.1
dns-server 8.8.8.8
!
iox
app-hosting appid centos
app-vnic gateway0 virtualportgroup 1 guest-interface 0
6
Install and
IOx App Hosting host the App
Install & Active App
app-hosting install appid centos package usbflash0:centosxe.tar
app-hosting activate appid centos
app-hosting start appid centos
IOx App Hosting
Access App
C9300# show app-hosting list

App id State
------------------------------------------------------
App1 RUNNING
C9300#
C9300# app-hosting connect appid App1 console

Connected to appliance. Exit using ^c^c^c
Get Your Hands Dirty
Service Containers – Legacy way of App Hosting
Container Network Model
Shared Dedicated
Applications inside the container appear as applications Applications inside the container appear as appliances on a
running natively on the host subnet reachable from the host
Examples: Nexus 3k, 9k, 6k, 7k, NCS xK Examples: C9K, ASR 1k, CSR 1kv, ISR4k, ISR 819
Container 1 Container 2 Container 1 Container 2

Network namespace: Host Network namespace: Host Network namespace: N1 Network namespace: N2
Container interfaces Container interfaces Container interfaces Container interfaces
eth0 eth1 eth2 eth0 eth1 eth2 veth0 veth1 veth2 veth0 veth1 veth2
Shared namespace: Linux Bridge

Interfaces are directly mapped to container
VPG
Multiple bridges and
virtual topologies possible
Host platform Host platform Forwarding Plane
Network namespace: Host Network namespace: Host
eth0 eth1 eth2 eth0 eth1 eth2
Physical interfaces Physical interfaces
Network Model
• VirtualPortGroup interface created on hosting device
Routing • vEth interface of application is connected to VPG
Model • Dedicated IP subnet for each hosted application
Unnumbered • VirtualPortGroup interface created on hosting device

• Unnumbered Interface of outgoing interface
Model • Saves one IP Address
Shared-IP • No dedicated subnet for the hosted application.

Model • Shares the IP/MAC of outgoing interface
Layer2 • Hosting device provides L2 connectivity to the

Model application
• Applicable for Cat9k platforms
Network Model – Routing Model
!
interface VirtualPortGroup0
Container
ip address 10.1.1.1 255.255.255.0
! vEth
10.1.1.2/24
!
app-hosting appid guestshell
guest-ipaddress 10.1.1.2 netmask 255.255.255.0
app-default-gateway 10.1.1.1 guest-interface 0
end 10.1.1.1/24
Virtual
PortGroup
WAN IOS-XE
172.16.1.1/24
Network Model – Unnumbered Model
!
interface Gigiabit0/0/0
ip address 10.1.1.1 255.255.255.0 Container
!
interface VirtualPortGroup0 vEth
ip unnumbered G0/0/0
! 10.1.1.2/24
!
app-hosting appid guestshell
app-default-gateway 10.1.1.1 guest-interface 0
end 10.1.1.1/24
Virtual
PortGroup
WAN IOS-XE
Gig0/0/0
10.1.1.1/24
Network Model – Shared-IP Model
!
interface GigiabitEthernet 1
ip address 10.1.1.1 255.255.255.0 Container
!
virtual-service ciscolive
ip shared host-interface GigabitEthernet1 vEth
!
10.1.1.1/24
csr1000v-2#
csr1000v-2# virtual-service install name ciscolive package flash:ciscolive.ova
Routing
virtual-service not supported
in Cat9000 platforms
WAN IOS-XE
Gig1
10.1.1.1/24
Network Model - Layer 2
Network Model – Layer 2 Model
VLAN Access
Container
!
interface GigabitEthernet1/1/2
vEth
switchport access vlan 101
!
10.1.1.2/24
app-hosting appid ciscolive

app-vnic AppGigEthernet vlan-access
vlan 101 guest-interface 0
Access
AppGig
Ethernet
10.1.1.1/24
Access L2Bridge
Gig1/1/2 IOS-XE
Network Model – Layer 2 Model
Trunk Mode
Container
!
interface GigabitEthernet1/1/2
vEth
switchport mode trunk
!
10.1.1.2/24
app-hosting appid ciscolive

app-vnic AppGigEthernet trunk
guest-interface 0
Trunk
AppGig
Ethernet
10.1.1.1/24
Trunk L2Bridge
Gig1/1/2 IOS-XE
Containers in
Cisco NX-OS
Containers in Cisco NX-OS
• Containers for NX-OS (Standalone)
• Open Agent Containers (OAC)
• Docker Container in NX-OS
• NX-OS: Container Orchestration with Kubernetes
• Use-cases
Container for NX-OS Standalone Switches
• Independent application
running on a Nexus switch
• Enables users to run any
application, including 3rd
party applications
• Run applications without
compromising security
Containers for Network Elements
• Virtualized environment to host Customer ISV* Cisco
“applications” on a Nexus device Apps Apps Apps
• Application Examples
• Cisco Virtual Services:
App1 App1
Linux shell: Guest shell & OAC (LXC) v1.17 v2.13
App 2 App 3
• Third Party Services: Container Container Container

Puppet Plugin, Chef Plugin (LXC),
Splunk universal forwarder plugin
• Troubleshooting: NX-OS
TCP dump, Custom monitoring agents,
Packet generation for real scenario testing
* Independent Software Vendors
Open Agent Container vs. Guest/Bash Shell
Open Agent Container (OAC) Guest / Bash Shell
NX-OS 7.3(0)D1(1) for Nexus 7xxx, NX-OS 7.0(3)I2(1) N3xx / N9xxx

7.3(0)N1(1) for Nexus 6xxx / N56xx
32/64-bit binary support, CentOS 7
environment, Wind River using Linux 3.x
32-bit binary support, CentOS 6.8
environment, Wind River using Linux 3.x
Access to /bootflash
Aware of which VDC the container is in Guest Shell Resize
Tune resource usage (Memory, Storage)

Inband traffic tunneled from Container to
“Netstack” to support VRF level front-panel Access to front-panel interfaces directly from
port access container “kstack”
Take-Away: All Nexus Platforms Support Containers
Front-Panel Ports in OAC
Netstack vs. Kstack – Quick Review
• Not every front-panel port in Nexus7000/Nexus6000 is represented
in OAC container
• A custom Netstack mechanism is used to access VRFs within the
VDC
• OAC still has IP reachability to Management and In-band VRFs but
cannot address individual interfaces
• There is no IP access to VRFs outside the VDC in which OAC is
deployed
• Kstack is not VDC aware as N3K/N9K don’t support VDCs. It runs in
default VDC (VDC 1) and provides access to all front-panel ports.
• Use-cases
Open Agent Container
Installation Steps
• Download oac.ova image from Cisco.com to your Nexus /bootflash
• Decide which VDC you want to install OAC into
• Enable NX-API feature – take advantage of this capability
• Install oac.ova and activate it via virtual-service command
Nexus-7k# feature nxapi
Nexus-7k# dir bootflash: | grep ova Takes 3-5 minutes. Be patience.
66355200 Apr 04 14:42:44 2019 oac.ova
Nexus-7k# virtual-service install name oac package bootflash:oac.ova

Note: Installing package 'bootflash:/oac.ova' for virtual service 'oac‘.
Once the install has finished, the VM may be activated. Use 'show virtual-service list' for progress
Nexus-7k(config)# virtual-service oac

Nexus-7k(config-virt-serv)# activate
Note: Activating virtual-service 'oac', this might take a few minutes. Use 'show virtual-service
list' for progress
OAC Activation
Troubleshooting Failure
Nexus-7k# show virtual-service list
Virtual Service List:

Name Status Package Name
-----------------------------------------------------------------------
oac Activate Failed oac.ova
Use internal event-history to investigate

Nexus-7k# show system internal virtual-service event-history debug
184) Event:E_VMAN_MSG, length:124, at 47795 usecs after Tue Apr 18 20:36:55 2017
(info): Response handle (nil), string Disk storage request (500 MB) exceeds remaining disk space
(344 MB) on storage
185) Event:E_VMAN_MSG, length:74, at 47763 usecs after Tue Apr 18 20:36:55 2017
(debug): Sending Response Message: Virtual-instance: oac - Response: FAIL
• Event-histories - are “running debugs” on by default (per vdc / per feature)

• No more waiting for maintenance windows to debug 
• No CPU Impact
OAC – A Closer Look
Nexus-7k# show virtual-service detail
Virtual service oac detail
State : Activated
Package information
Name : oac.8.1.1.ova
Path : bootflash:/oac.8.1.1.ova
Application
Name : OpenAgentContainer
Installed version : 1.0
Description : Cisco Systems Open Agent Container
Signing
Key type : Cisco release key
Method : SHA1
Licensing
Name : None
Version : None
Resource reservation
Disk : 500 MB
Memory : 384 MB
CPU : 1% system CPU
Attached devices
Type Name Alias
---------------------------------------------
Disk _rootfs
Disk /cisco/core
Serial/shell
Serial/aux
Serial/Syslog serial2
Serial/Trace serial3
OAC Shell
Console Access
• Attaching to the container Nexus-7k# virtual-service connect name oac console
Connecting to virtual-service. Exit using ^c^c^c
console you may login as Trying 127.1.1.5...
oac/oac or root/oac and

Connected to 127.1.1.5.
Escape character is '^]'.
change the root password. CentOS release 6.8 (Final)
Logging in as oac/oac and Kernel 3.14.39ltsi++ on an x86_64
(none) login:
sudo to root is also possible.
• Creating users and changing passwords in the container
does not impact other users in the switch due to the kernel
namespace isolation.
• Currently no built-in equivalent to ssh over loopback. Start
sshd in the container, as console may be slow.
OAC Shell
Enabling SSH Access
• Shared interface with NX-OS environment
• sshd must listen on non-standard TCP port, typically 2222 (as port
22 already taken by the Management interface for NXOS)
• sshd must also be started in the VRF required to bind to the correct
interfaces
• Modify /etc/ssh/sshd_config and start sshd
[root@(none)]# sudo cat /etc/ssh/sshd_config [root@(none)]# chvrf management /etc/init.d/sshd start
Port 2222 Starting sshd:
#AddressFamily any [ OK ]
ListenAddress 0.0.0.0
#ListenAddress :: [root@(none)]# vrf2num management
2
Added config to pre-set VRF
at the initialization / startup [root@(none)]# more /etc/init.d/sshd | grep DCOS
export DCOS_CONTEXT=2
OAC Shell
Setup Network Access Nexus-7k# show ip internal info interface mgmt 0
• DNS IP extension for iod 131/mgmt0
lcache 0x5a86776c, Ext 0x5a86776c, if_private 0xd7b4c2fc,
/etc/resolv.conf maddresses (nil), iod 131
nameserver 8.8.8.8 VRF 2/management admin up 1
search foo.com unnumbered if : 0 (null)
ip_ctxt 0xdf928e34, ip_vaddr (nil) (0),
directed_broadcast_acl (nil)Total 'ip forward' configured
interface count: 0
• Proxy?
/etc/environment [root@(none) ~]$ chvrf management
export http_proxy=http://proxy.foo.com:80 [root@(none) ~]$ getvrf
export https_proxy=https://proxy.foo.com:80 Management
[root@(none) ~]$
• VRF Access (chvrf, getvrf, and vrf2num) [root@(none) ~]$ vrf2num management
vrf2num <vrf_name> 2
[root@(none) ~]$
chvrf vrf_name <cmd>
chvrf vrf_name
OAC Shell
Running Host CLI
• The dohost command is a Python wrapper script around NX-API
calls using Linux domain sockets back to NX-OS
[root@(none) ~]$ dohost "show clock"
23:03:42.122 UTC Wed May 01 2019
Time source is NTP
[root@(none) ~]$
• show commands and configuration access within the VDC the

container is created
• Enhancement coming for dohost to work outside the default VDC
Package Management
Yum Install
• Install the packages you want, just like any CentOS 6.8 Linux
environment
• Ensure you callout the correct VRF to access Yum repository or have
environment variable setup already
• Also make sure any proxy [root@(none) ~]$ chvrf management
[root@(none) ~]$ sudo yum install -y vim
server and DNS configuration Loaded plugins: fastestmirror
base | 3.6 kB 00:00
is applied for connectivity extras
updates
| 3.4 kB
| 3.4 kB
00:00
00:00
(1/4): base/7/x86_64/group_gz | 166 kB 00:00
• No default route is needed in (2/4): base/7/x86_64/primary_db | 6.0 MB 00:03
(3/4): extras/7/x86_64/primary_db | 188 kB 00:03
Linux as we are using the (4/4): updates/7/x86_64/primary_db | 4.2 MB 00:04
native Netstack VRF routing

Determining fastest mirrors
Setting up Install Process
table Resolving Dependencies

<snip>
Package Management
Python PIP: Install
• Python’s package manager is called PIP
• Users can install optional libraries (Py Packages)
• May require specifying proxy server
• Bootstrap PIP with get_pip.py
https://pip.pypa.io/en/stable/installing/
pip install requests -–proxy=proxy.foo.com:80
OAC Persistence
Supervisor Engine Failover/Reload
• Containers on Standby Supervisor is offline (inactive). Switchover
triggers activation in new Active Supervisor (boot up)
• Agents/Scripts must be scripted to bring up any functionality that
must come up on startup of the container
• Run copy run start to ensure container filesystem changes are
persisted to flash
• On N7000/N7700 a reload of the VDC containing OAC container
will cause the container to go offline and be re-activated on VDC
bring-up
Guest Shell and OAC
Guest Shell and OAC
• Guest Shell supports guestshell sync to command to synchronize
container filesystem to Standby Supervisor
• Currently OAC does not replicate container filesystem between
supervisors
• Looking at EXT4 capabilities to do real-time replication of container
state between Supervisors
• Container on Standby supervisor is non-active - On switchover
event the container will be activated (booted)
Shared Network Interfaces
Interfaces existing in multiple containers or native environment
• It all comes back to Network Shared Network Model
Namespaces isolating, among
other things, sockets and file Container 1
Network namespace: Host
Container 2
descriptors into each container Container interfaces Container interfaces
• Once a socket is listening on a

eth0 eth1 eth2 eth0 eth1 eth2
port, the kernel tracks those

structures by Namespace. Shared namespace:
Interfaces are directly mapped to container
• As a result the kernel knows

how to direct traffic to the Host platform – Kernel Space
correct container socket

eth0 eth1 eth2
Physical interfaces
• Use-cases
Docker Container in NX-OS
Prerequisites
switch# conf t
1. Enable bash-shell feature switch(config)# feature bash-shell
Variables must be set up in /etc/sysconfig/docker.
2. Set up proxy (if applicable) Example:

root@switch# cat /etc/sysconfig/docker | grep http
export http_proxy=http://<server>:<port>
export https_proxy=http://<server>:<port>
switch(config)# vrf context management
3. Set up domain name switch(config-vrf)# ip domain-name <domain-name>

switch(config-vrf)# ip name-server <ip-addr>
and DNS root@switch# cat /etc/resolv.conf

domain <domain-name>
nameserver <ip-addr>
root@switch#
Docker Service “service docker stop”
to stop the service
• Start Docker Service as a Load Bash and become superuser.
switch# run bash sudo su -
Superuser in the bash- Start the Docker daemon.
shell root@switch# service docker start
• Check the status root@switch# service docker status

dockerd (pid 3597) is running...
root@switch#
• Dockerpart …
Once you start the Docker daemon, a
switch# dir bootflash:dockerpart
dockerpart file will be created in the 2000000000 Mar 19 12:50:14 2019 dockerpart
bootflash: with 2GB memory reserved.
Do not delete or tamper it, as it is critical
to the docker functionalities.
Docker Container in NX-OS Load specific volume
Docker Container root@switch# docker run --name=myalpine -v

/var/run/netns:/var/run/netns:ro,rslave --rm --network host --cap-add
SYS_ADMIN -it alpine
/ # ip netns list
• Start Docker Container management
default You are in Alpine!
/ # ifconfig –a
Name: alpinerun <snip>
docker0 Link encap:Ethernet HWaddr 02:42:45:CA:4F:58
Network Mode: Host inet addr:172.17.0.1 Bcast:0.0.0.0 Mask:255.255.0.0
UP BROADCAST MULTICAST MTU:1500 Metric:1
Image: Alpine Linux RX packets:1136 errors:0 dropped:0 overruns:0 frame:0
TX packets:1816 errors:0 dropped:0 overruns:0 carrier:0
• Network Interfaces collisions:0 txqueuelen:0
RX bytes:85162 (83.1 KiB) TX bytes:2437459 (2.3 MiB)
Docker bridge eth1 Link encap:Ethernet HWaddr 00:DE:FB:FB:50:E0

network Interface inet addr:10.122.140.113 Bcast:10.122.140.127
Mask:255.255.255.192
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2904819 errors:0 dropped:0 overruns:0 frame:0
Shared Interface. TX packets:1168176 errors:0 dropped:0 overruns:0 carrier:0
Same as Switch’s collisions:0 txqueuelen:1000
Mgmt0 interface RX bytes:527940304 (503.4 MiB) TX bytes:114204131 (108.9
MiB)
<snip>
Qualities Of Containers
• Dedicated namespaces for myalpine myalpine1
containers
Process Process Process Process Process
• Hostname, Process, IPC, File
System, Network, User
(subsets are possible)
Process Process Process Process Process
• Resource Isolation between
container and kernel Container Container
File File File System
(cgroups). System System
• Shared Kernel
Kernel
Docker Container – Status & Persistence Through Reload/Sup Switchover
root@switch# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
47d5c0a2916b alpine "/bin/sh" 56 mins ago Up 56 mins myalpine
52e7a1e5731a alpine "/bin/sh" 31 mins ago Up 31 mins myalpine2
root@switch#
Docker containers root@switch# chkconfig --add docker
in action! root@switch#
Starts an Alpine
container and
• Configure the switch root@switch# chkconfig --list | grep docker
docker 0:off 1:off 2:on 3:on 4:on 5:on 6:off configures it to
always restart
to restart docker root@switch#
unless it is explicitly
service (or not) at root@switch# chkconfig --del docker
root@switch# chkconfig --list | grep docker
stopped or docker is
restarted
each reload root@switch#
root@switch# docker run -dit --restart unless-stopped alpine
• Make sure to copy the /bootflash/dockerpart to standby sup in the

redundant supervisor scenario
• Use-cases
Container Orchestration Node 1
Kubernetes Components
Kubelet
Master Kube-Proxy
Kube-Controller Cloud-Controller
Manager Manager Container Node 2
Runtime
Kubelet
Kube API-Server
etcd Kube-Proxy
Kube Scheduler
Container
Runtime
Container Orchestration
Kubernetes and NX-OS – What do we have so far?
Control-Plane
Inband Channel
Supervisor Docker Container
NX-OS
Inband port
myalpine Ready to register Nexus9000 worker

node to Kubernetes Master??
Packets sent to the CPU

Docker
eth1 Service
Bash
Kernel
Eth1-1 Eth1-2 ... Eth1-N Mgmt0
Physical interfaces Nexus9000
Docker & Kubernetes in Master Node
Make sure Docker service cisco@server-1:~$ sudo -i

[sudo] password for cisco:
is started and running root@server-1:~# sudo -
root@server-1:~$ service docker start
docker run -d \
Start etcd --net=host \
component of
gcr.io/google_containers/etcd:2.2.1 \
/usr/local/bin/etcd --listen-client-urls=http://10.0.0.6:4001 \
Kubernetes
--advertise-client-urls=http://10.0.0.6:4001 --data-dir=/var/etcd/data
docker run -d --name=api \

--net=host --pid=host --privileged=true \
Run API server of gcr.io/google_containers/hyperkube:v1.2.2 \
/hyperkube apiserver --insecure-bind-address=10.0.0.6 \
Kubernetes --allow-privileged=true \
--service-cluster-ip-range=172.16.1.0/24 \
--etcd_servers=http://10.0.0.6:4001 --v=2
Docker & Kubernetes in Master Node (Contd.)
docker run -d --name=kubs \
--volume=/:/rootfs:ro --volume=/sys:/sys:ro --volume=/dev:/dev \
--volume=/var/lib/docker/:/var/lib/docker:rw \
--volume=/var/lib/kubelet/:/var/lib/kubelet:rw \
Run kubelet --volume=/var/run:/var/run:rw --net=host --pid=host \
component of
--privileged=true \
gcr.io/google_containers/hyperkube:v1.2.2 \
Kubernetes
/hyperkube kubelet --allow-privileged=true \
--hostname-override="10.0.0.6" \
--address="10.0.0.6" --api-servers=http://10.0.0.6:8080 \
--cluster_dns=10.0.0.10 \
--cluster_domain=cluster.local --config=/etc/kubernetes/manifests-multi
Run proxy docker run -d --name=proxy \

--net=host --privileged \
component of gcr.io/google_containers/hyperkube:v1.2.2 \
Kubernetes
/hyperkube proxy --master=http://10.0.0.6:8080 --v=2
Kubernetes in Nexus – Kubernetes Worker
root@switch# docker run -d --name=kubs \
--net=host --pid=host --privileged=true \
--volume=/:/rootfs:ro --volume=/sys:/sys:ro --volume=/dev:/dev \
--volume=/var/lib/docker/:/var/lib/docker:rw \
--volume=/var/lib/kubelet/:/var/lib/kubelet:rw \
Join Nexus9000 to the --volume=/var/run:/var/run:rw \
pod by registering to /hyperkube kubelet --allow-privileged=true --containerized \
--enable-server --cluster_dns=10.0.0.10 \
the Master --cluster_domain=cluster.local \
--config=/etc/kubernetes/manifests-multi \
--hostname-override="10.0.0.6" \
--address=0.0.0.0 --api-servers=http://10.0.0.6:4001
root@switch# docker run -d --name=proxy \
--net=host --privileged=true \
/hyperkube proxy --master=http://10.0.0.6:4001 --v=2
NX-OS and Kubernetes – What do we have so far?
Nexus9000
Control-Plane
K8s Master
Inband Channel Kube-Controller Cloud-Controller

Supervisor Manager Manager
NX-OS
Inband port
K8s Node
Docker Container Kube API-Server
Kubelet
myalpine Kube-Proxy
Packets sent to the CPU etcd
Container Kube Scheduler
Runtime
10.0.0.6/24 NIC
Docker Service
eth1 Ubuntu Server
Eth1-1 Eth1-2 ... Eth1-N Bash
Physical interfaces Kernel Network
10.102.242.131/28
Mgmt0
• Use-cases
Use-cases
Management & Troubleshooting At Scale
• Agents can be used to monitor each and every switch’s
parameters, config, packet counters or log events.
Reporting can be local via web server running in the
container and/or some remote data collection server
(Syslog, Splunk, Elastic Search, local Web Server). Sky is
• Agents can poll the switch frequently to support a more
consistent, less error prone operational environment.
the
• Distributed agents co-located on switches scale higher
limit!!
than centralized server, automating hundreds to
thousands of switches.
• Ease of managing switches at scale. Original use-case
for containers was for Puppet and Chef integration.
Use-case 1: Develop a custom CLI
to track end-host mobility
NXOS Software Development Kit (SDK)
Introduction
• Simple, flexible and powerful tool for
custom on-the-box applications Custom Applications
(Python, C++)
Native NX-OS
Applications
running in a Docker container – to

Linux
gain access to NX-OS infrastructure NXOS Infra SDK
Abstraction Layer Library
• Languages supported: Python or C++

CLIs
HA
• Run natively. Startup and
management handled by NX-OS Syslog, Events
& Faults
DME etc.
• Define your own custom CLIs,

syslogs, events and more Event
Manager
NX-OS
• Supported from 7.0(3)I6(1) onwards Nexus9000
Tracking End-Hosts Mobility
Building Custom Application
Application: Track movement of an end-host

Nexus93180
Switch: Nexus9000 C93180LC-EX
NX-OS: 7.0(3)I7(1)
Capability used: VSH
Eth1/3 Eth1/5
Eth1/4 Language: Python
Procedure followed:
1) Build the application in a standalone host running CentOS 7
Network (which can also be done NXOS Bash Shell, using native Python
capability)
2) Pull Docker container and setup NX-SDK environment
3) Build RPM package of the Python App
4) Transfer the RPM package to Nexus switch, install and activate
5) Verify the Service
6) Use custom application’s CLI to track end-host(s)
End-to-End Process Flow
BASH:
Python SDK Environment not needed.
Setting up the C++ SDK Environment is optional, but recommended.
ENXOS SDK VSH:
Environment SDK environment is mandatory. Apps must be built as an RPM package,
and installed as a package.
Building an (1) Docker pull of ENXOS SDK Build environment

application (2) Start/Run Docker container.
using (3) [optional] Upgrade/Downgrade NX-SDK using git pull or git clone
programming (4) For C++, add the application to the Makefile and make sure builds are
languages* error-free. For Python, nothing is required.
Generate RPM package, using built-in rpm_gen.py script.

Packaging the
For complex applications, manually generate RPM package following steps
application
provided here.
* Currently Python and C++ supported. Support for more languages in future road-map.
End-to-End Process Flow
Copy the App (binary) or RPM package to the switch’s bootflash:

VSH:
Installing the
Add package to the installer with “install add bootflash:<file.rpm>”
application in
and activate with “install activate bootflash:<file.rpm>” command.
Switch
BASH:
Install RPM package with “yum install /bootflash/<file.rpm>”
VSH:
Start the application with “nxsdk service-name <app-name>”. If App
Running the
is installed at non-default location, then do “nxsdk service-name
application in
<full-path/app-name>”
Switch
BASH:
In the switch config, “run bash sudo su”, and then “app-full-path &”
1) Building a Python application – CLIs
def get_mac_from_arp(cli_parser, clicmd, target_ip):
exec_cmd = "show ip arp {}".format(target_ip)
arp_cmd = cli_parser.execShowCmd(exec_cmd, nx_sdk_py.R_JSON)
if arp_cmd:
try:
Check ARP arp_json = json.loads(arp_cmd)
except ValueError as exc:
and get return None
MAC-addr count = int(arp_json["TABLE_vrf"]["ROW_vrf"]["cnt-total"])
if count:
intf =arp_json["TABLE_vrf"]["ROW_vrf"]["TABLE_adj"]["ROW_adj"]
if intf.get("ip-addr-out") == target_ip:
target_mac = intf["mac"]
def get_vlan_from_cam(cli_parser, clicmd, target_mac):

exec_cmd = "show mac address-table address {}".format(target_mac)
mac_cmd = cli_parser.execShowCmd(exec_cmd, nx_sdk_py.R_JSON)
From MAC if mac_cmd:
try:
entry, find cam_json = json.loads(mac_cmd)
the current except ValueError as exc:
Interface return None
mac_entry = cam_json["TABLE_mac_address"]["ROW_mac_address"]
if mac_entry:
<snip>
1) Building a Python application – CLIs (cont’d)
def find_mac_movement(cli_parser, clicmd, target_mac, mac_vlan):
exec_cmd = "show system internal l2fm l2dbg macdb address {} vlan
{}".format(target_mac, mac_vlan)
l2fm_cmd = cli_parser.execShowCmd(exec_cmd)
if l2fm_cmd:
event_re = re.compile(r"^\s+(\w{3}) (\w{3}) (\d+) (\d{2}):(\d{2}):(\d{2})
(\d{4}) (0x\S{8}) (\d+)\s+(\S+) (\d+)\s+(\d+)\s+(\d+)")
unique_interfaces = []
l2fm_events = l2fm_cmd.splitlines()
for line in l2fm_events:
res = re.search(event_re, line)
Check the if res:
end-host day_name = res.group(1)
month = res.group(2)
movement day = res.group(3)
hour = res.group(4)
minute = res.group(5)
second = res.group(6)
year = res.group(7)
if_index = res.group(8)
db = res.group(9)
event = res.group(10)
src = res.group(11)
slot = res.group(12)
fe = res.group(13)
2-3) Setting up Docker environment and Building RPM package
2) Install Docker, pull NX-SDK Docker container and run
[root@localhost ~]# yum -y install docker
[root@localhost ~]# docker pull dockercisco/nxsdk:v1
[root@localhost ~]# docker run –it dockercisco/nxsdk:v1 /bin/bash
root@b7d33ce8a7b8:/# cd /NX-SDK
root@b7d33ce8a7b8:/NX-SDK# git pull
3) Copy the Python App and build RPM

root@b7d33ce8a7b8:/# cd /root
root@b7d33ce8a7b8:~# mkdir nxsdk-scripts
root@b7d33ce8a7b8:~# cd nxsdk-scripts/
root@b7d33ce8a7b8:~# cp /bootflash/ip_move.py .
root@b7d33ce8a7b8:~/nxsdk-scripts# python /NX-SDK/scripts/rpm_gen.py ip_move.py –s
/root/nxsdk-scripts –u
<snip>
RPM package has been built
SPEC file: /nxsdk/rpm/SPECS/ip_move.py.spec

RPM file: /nxsdk/rpm/RPMS/ip_move.1.0-1.5.0.x86_64.rpm
4-5) Installing RPM in Nexus, Activate and Verify Service
4) Move the RPM to Nexus, install and activate
N93180# copy ftp://<server>/ip_move.1.0-1.5.0.x86_64.rpm bootflash: vrf management
N93180# install add bootflash:ip_move.1.0-1.5.0.x86_64.rpm
N93180# install activate ip_move.1.0-1.5.0.x86_64
5) Enable NX-SDK feature, activate and verify the service

N93180(config)# feature nxsdk
N93180(config)# nxsdk service-name ip_move.py
N93180# show nxsdk internal service

NXSDK Started/Temp unavailable/Max services : 0/0/32
NXSDK Default App Path : /isan/bin/nxsdk
NXSDK Supported Versions : 1.0
Service-name Base App Started(PID) Version RPM Package
--------------------- --------------- ------------ ------- ------------
/isan/bin/ip_move.py nxsdk_app1 VSH(28161) 1.0 ip_move.py-1.0-1.5.0.x86_64
6) Using the Service
N93180# show ip_move.py 20.20.20.3
20.20.20.3 is currently present in ARP table, MAC address 0010.9400.0002
0010.9400.0002 is currently present in MAC address table on interface Ethernet1/3, VLAN 20
0010.9400.0002 has been moving between the following interfaces, from most recent to least
recent:
Fri Apr 20 12:05:17 2019 - Ethernet1/3 (Current interface)
Fri Apr 20 12:04:13 2019 - Ethernet1/5
Fri Apr 20 12:04:13 2019 - Ethernet1/4
Fri Apr 20 12:03:50 2019 - Ethernet1/5
Fri Apr 20 12:03:50 2019 - Ethernet1/4
Fri Apr 20 12:03:26 2019 - Ethernet1/5
Use-case 2: Running a containerized
App – Anomaly Detector
Nexus 9000 – Anomaly Detector
Steps: Building and Running a Containerized-Application
1. Linux Python Integrated Development Environment (IDE) –
Develop a Python code with core functions
2. GitHub - Build a Dockerfile, to set environment variables, install
requirements and execute the Python code.
3. DockerHub - Build a containerized application
4. Nexus9000 - In the Bash shell, under the management
namespace, build a docker-compose file (e.g., docker-
compose.yml) and execute the App.
Nexus 9000 -Anomaly Detector
Building and Running a Containerized-Application
3
Control-Plane 4
Floodlight
(Containerized App)
Inband Channel Floodlight
Supervisor NameSpace: Management DockerHub
NX-OS
Python
Inband port Docker Pull
/startup-config
/var/log
/bootflash
2
1
Container
eth1 Floodlight
(Dockerfile, Python
Packets sent to the CPU & Requirements)
Bash
Kernel GitHub
Eth1-1 Eth1-2 ... Eth1-N Linux Host
[Python Integrated
Physical interfaces Nexus9000 Development Environment]
Floodlight
Floodlight Application
(Dockerfile,
Requirements &
2 Python)
GitHub Repository – Dockerfile & Requirements GitHub

Required TCPDUMP
Off latest Alpine and Tshark Libraries
Application requirements
Dockerfile Execution !!
Requirements
Reference: Floodlight - GitHub Repository
Floodlight
(Dockerfile,
Requirements &
2 Python)
GitHub Repository – Python code GitHub

Capture the traffic
Used Scapy to read the packets in a PCAP file
Check the features

enabled and configs
to synthesize filters
Build the filters (based on

the startup-config)
Sample: OSPF
Floodlight
(Dockerfile,
Requirements &
2 Python)
GitHub Repository – Python code GitHub

Filter the packets for traffic not expected to be at the CPU, and summarize
Apply the filter synthesized, to

identify the packets that are
not expected to be at the CPU
Floodlight
Floodlight Application 3
(Containerized App)
DockerHub
DockerHub Repository
Connected to GitHub repository
Containerized App is available at:

DockerHub - Floodlight Repository
Floodlight
Floodlight Application 4 Bash
Nexus9000 – Bash Shell and Docker Nexus9000
enable Bash shell
N93180(config)# feature bash-shell root@N93180# cd floodlight/
N93180(config)# end root@N93180# ls -l
N93180# run bash sudo su – run Bash shell
-rw-r--r-- 1 root root 316 Jan 16 14:24 docker-compose.yml
root@N9380# root@N93180#
root@N9380# ip netns exec management bash root@N93180#
root@N93180# more docker-compose.yml
version: "3"
pulls the latest image
services: from the dockerhub
Namespace: Management floodlight:
Make sure it has internet image: chrisjhart/floodlight:latest
connectivity, if DockerHub is used container_name: floodlight
volumes:
- /var/sysmgr/startup-cfg/ascii/system.cfg:/startup-config
- /var/log/:/var/log/
Mount the - /bootflash:/bootflash
required volumes environment:
- DEBUG=1
Reference: Installing - EXPORT=/bootflash/example_pcap.pcap
Docker Compose in NX- network_mode: "host"
OS Bash Shell root@N93180# Container runs
in “host” mode
Floodlight
Floodlight Application That’s your App !!
CLI executes docker-compose.yml 4 Bash
Nexus9000 – Bash Shell and Docker Nexus9000
root@N93180# docker-compose up
Docker Starting floodlight ... done
Control-Plane Hub Attaching to floodlight
floodlight INFO [LOG] Debug logging level set!
floodlight INFO [SETUP] NX-OS startup-config file detected
Packet sent
floodlight INFO [FILTER] OSPF feature and configuration found!
to the CPU Docker floodlight INFO [FILTER] HSRP configuration not found, skipping...
Pull <snip>
Inband port
floodlight INFO ==== FILTERS ====
floodlight | ‘ip': ['224.0.0.5', '224.0.0.6'],
Floodlight floodlight | 'ip_protocol_type': ['89’],
NameSpace: Management <snip>
floodlight | 'protocols': ['OSPF', 'BGP', 'Spanning Tree Protocol', 'SSH',
Python 'CDP', 'LLDP']}
floodlight INFO [CAPTURE] Beginning packet capture, be back in 60 seconds...
/startup-config floodlight INFO [CAPTURE] Packet capture finished! 259 packets in capture
/var/log floodlight INFO [UNEXPECTED] Number of unexpected packets: 138
/bootflash floodlight INFO ===== RESULTS =====
floodlight INFO 14,879 bytes (123 packets) | TCP (TCP )
00:01:02:03:04:05 10.150.53.63:50449 -> 10.150.53.229:2345 00:de:fb:fa:64:c7
Container <snip>
eth1 floodlight INFO [WRITE-PCAP] Successfully wrote unexpected packets to PCAP
Bash at /bootflash/example_pcap.pcap
Nexus9000 floodlight exited with code 0
root@N93180#
Container Use-cases
Operations
• Inventory Management
• Hardware Uptime Check
• Scalability Check Sky is
• Control-plane Health Check
the
Configuration Consistency Check
•
limit!!
• Traffic Profiling and Top-Talkers
Summary
&
Take-Aways
• Motivation behind this session
Summary • Topics to discuss
• Goals and potential learnings
• Introduction
• Refresh on virtualization and container concepts
• How these are applicable to Cisco’s platforms
• Virtualization and Containers
• Detailed discussion on different container
capabilities in IOS-XE platforms
• Procedure to enable container and App
• Containers in Cisco IOS-XE hosting in IOS-XE
• Discussion on various networking models
• Containers in Cisco NX-OS • Detailed discussion on different container
capabilities like Open Agent Container (OAC)
• Deploying Docker in NX-OS and orchestrate
it with Kubernetes
• Use-cases: How to bring them to your day-
to-day operations
Take-Aways …
Architecture and capabilities of Cisco IOS-XE
and NX-OS are innovative to virtualize the
infrastructure and host applications
Rich set of features and software development

environment make the automation efforts and
integration to the workflow seamless
Wealth of knowledge shared in this session

enables and empowers each one of you to
make progress in the digitization journey
Complete your
online session • Please complete your session survey
evaluation after each session. Your feedback
is very important.
• Complete a minimum of 4 session
surveys and the Overall Conference
survey (starting on Thursday) to
receive your Cisco Live water bottle.
• All surveys can be taken in the Cisco Live
Mobile App or by logging in to the Session
Catalog on ciscolive.cisco.com/us.
Cisco Live sessions will be available for viewing
on demand after the event at ciscolive.cisco.com.
Continue your education
Demos in the
Walk-in Labs
Cisco campus
Meet the engineer

Related sessions
1:1 meetings
Appendix
Guest Shell and Bash Shell in NX-OS
Guest and Bash Shell
Hosting Environments
Open NX-OS provides two environments for hosting 3rd-Party applications or Agents/Clients.
• Bash Shell (Native):

• Third-Party applications built for WRL distribution can be installed natively on the
WRL root file system.
• Agent installed in this environment can manipulate native NX-OS packages, files
and services.
• Guest Shell:
• Third-Party applications built for CentOS7 can be installed within the secure
Guest Shell environment. For additional details please refer to Guest Shell user
guide.
• Agent installed in this environment provides security against inadvertently
damaging the switch OS without sacrificing capabilities.
Guest Shell Applications
Linux Shell Environment
• Maintain NX-OS system integrity
• Isolated User Space
• Fault Isolation Linux
• Resource Isolation applications
• Integrate into your Linux workflow Guest Shell
• On-box rapid prototyping Open Agent Container

• Device-level API Integration API
• Scripting (Python / Ruby / Bash / JavaScript)
NX-OS
• Linux Commands
• Integrated with NX-OS
Enabling Guest Shell Guest shell is an OVA
N9K-C93180YC# show guestshell
Virtual service guestshell+ detail
Cent OS 7 environment State : Activated
Package information
N9K-C93180YC# guestshell enable
Name : guestshell.ova
Path : /isanboot/bin/guestshell.ova
2019 May 1 02:10:24 N9K-C93180YC %$ VDC-1 %$
Application
%VMAN-2-INSTALL_STATE: Installing virtual service
'guestshell+’ Name : GuestShell
Installed version : 2.4(0.0)
Description : Cisco Systems Guest Shell
2019 May 1 02:10:53 N9K-C93180YC %$ VDC-1 %$
%VMAN-2-INSTALL_STATE: Install success virtual <snip>
service 'guestshell+'; Activating Resource reservation
Disk : 250 MB
2019 May 1 02:10:53 N9K-C93180YC %$ VDC-1 %$ Memory : 256 MB
%VMAN-2-ACTIVATION_STATE: Activating virtual CPU : 1% system CPU
service 'guestshell+’
Attached devices
2019 May 1 02:11:26 N9K-C93180YC %$ VDC-1 %$ Type Name Alias
%VMAN-2-ACTIVATION_STATE: Successfully activated ---------------------------------------------
virtual service 'guestshell+’ Disk _rootfs
Disk /cisco/core
2019 May 1 02:11:26 N9K-C93180YC %$ VDC-1 %$ Serial/shell
%VMAN-2-GUESTSHELL_ENABLED: The guest shell has Serial/aux
been enabled. The command 'guestshell' may be used Serial/Syslog serial2
to access it, 'guestshell destroy' to remove it. Serial/Trace serial3
Guest Shell
Resizing the Resources Allocated
• You may want to resize your Guest Shell filesystem to a larger
footprint
• For example to Yum install full Linux development environment (gcc…) you
will need to resize container to a minimum of 1GB
chvrf management yum groupinstall "Developer Tools”
• Generally, it’s easier to build equivilent CentOS virtual machine off box to
develop and transfer compiled binaries over
• Also, maybe your script requires more memory or CPU?
Nexus-9k# guestshell resize rootfs 1500
Note: Please disable/enable or reboot the Guest shell for root filesystem to be resized
Nexus-9k# guestshell resize cpu 5

Note: Please disable/enable or reboot the Guest shell for system CPU share to be resized
Guest Shell
Access Shell and Run Scripts
[guestshell@guestshell ~]$ python hello.py
Hello, World!
One
N9K-C93180YC# guestshell Python 2.7.5 is packaged
Two
[admin@guestshell ~]$ pwd
Three
with guest shell
/home/admin
[admin@guestshell ~]$ su - guestshell [guestshell@guestshell ~] python
Password: Python 2.7.5 (default, Jun 17 2014, 18:11:42)
[guestshell@guestshell ~]$ pwd [GCC 4.8.2 20140120 (Red Hat 4.8.2-16)] on linux2
/home/guestshell Type "help", "copyright", "credits" or "license" for
more information.
[guestshell@guestshell ~]$ ls
>>> print "a"
hello.py
[guestshell@guestshell ~]$ cat hello.py
a [Control + D three
#!/usr/bin/env python
>>> times to exit ]
[guestshell@guestshell ~] $exit
import sys
N9K-C93180YC# run guestshell sudo python
print "Hello, World!"
/home/guestshell/hello.py
list = ['One', 'Two', 'Three']
Hello, World!
for item in list:
One
print item
Two
[guestshell@guestshell ~]$
Three
N9K-C93180YC#
Guest Shell
SSH and Console Access
• By default, the guest shell starts an Open-SSH server upon boot
up, and listening at port number 4022. Username/password less.
• Guest shell can also be accessed with virtual-service connect
name guestshell+ console command. This is helpful if SSH server
is killed, or configurations altered (available at
/etc/ssh/sshd_config).
• To have SSH server at custom VRF and port,
chvrf vrf_name /usr/sbin/sshd -p port_number
• Use NX-OS CLI “show socket connections” to find a free port to
use.
Guest Shell Management VRF
Internet
Mgmt0
Network Access
NX-OS
N9K-C93180YC# guestshell
[admin@guestshell ~]$ ping 8.8.8.8 Inband Channel
connect: Network is unreachable Sup eth1
Engine Inband port
[admin@guestshell ~]$ ifconfig lo
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 Default Mgmt
inet 127.0.0.1 netmask 255.255.0.0 Packets sent to the CPU VRF VRF
<snip> Shell
Kernel
[admin@guestshell ~]$ ifconfig veobc
veobc: flags=67<UP,BROADCAST,RUNNING> mtu 1494 Eth1-1 Eth1-2 ... Eth1-N Default VRF
inet 127.1.2.1 netmask 255.255.255.0 Physical interfaces Nexus9000
broadcast 127.1.2.255
<snip> [admin@guestshell ~]$ ifconfig eth1
[admin@guestshell ~]$ ip netns list eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>
management mtu 1500
default inet 10.102.242.131 netmask
[admin@guestshell ~]$ chvrf management 255.255.255.240 broadcast 10.102.242.143
[admin@guestshell ~]$ ping 8.8.8.8 ether 00:3a:9c:5a:00:60 txqueuelen
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data. 1000(Ethernet)
64 bytes from 8.8.8.8: icmp_seq=1 ttl=121 RX packets 717307 bytes 57542116 (54.8 MiB)
time=7.96 ms TX packets 101602 bytes 10472080 (9.9 MiB)
<snip> <snip>
Guest Shell
Networking
• Each front-panel port, VLAN interface and port-channel are
represented in Linux kernel as Net-devices using Cisco kstack
implementation
• ARP and Routing tables synchronized between NX-OS and Native
Linux Kernel via Netbroker module
• VRFs represented in Linux Kernel as first class Kernel Network
Namespaces
• This allows container direct access to network elements such as
tcpdump on a given front-panel port
Guest Shell
Routes and ARP Synchronization N9K-C93180YC# show run int mgmt0
<snip>
N9K-C93180YC# show ip route vrf management interface mgmt0
IP Route Table for VRF "management" vrf member management
<snip> ip address 10.102.242.131/28
0.0.0.0/0, ubest/mbest: 1/0 N9K-C93180YC#
*via 10.102.242.129, [1/0], 1w4d, static
10.102.242.128/28, ubest/mbest: 1/0, attached
*via 10.102.242.131, mgmt0, [0/0], 1w4d, direct
10.102.242.131/32, ubest/mbest: 1/0, attached
*via 10.102.242.131, mgmt0, [0/0], 1w4d, local
N9K-C93180YC# show ip arp vrf management
Address Age MAC Address Interface Flags
10.102.242.129 00:02:34 001e.f7be.70c2 mgmt0
[admin@guestshell ~]$ route

Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default GW1.cisco.com 0.0.0.0 UG 0 0 0 eth1
10.102.242.128 0.0.0.0 255.255.255.240 U 0 0 0 eth1
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
[admin@guestshell ~]$ arp -a
GW1.cisco.com(10.102.242.129) at 00:1e:f7:be:70:c2 [ether] PERM on eth1
Guest Shell
Front-Panel Ports
[admin@guestshell ~]$ ip link show

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: dummy0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN mode DEFAULT
link/ether 72:2c:90:bc:8d:59 brd ff:ff:ff:ff:ff:ff
<snip>
34: Eth1-1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN mode DEFAULT qlen 100
link/ether 00:3a:9c:5a:00:67 brd ff:ff:ff:ff:ff:ff
35: Eth1-2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN mode DEFAULT qlen 100
<snip>
87: Eth1-54:<NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN mode DEFAULT qlen 100
[admin@guestshell ~]$
[admin@guestshell ~]$ ifconfig Eth1-2
Eth1-2: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ether 00:3a:9c:5a:00:67 txqueuelen 100 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
Guest Shell
Front-Panel Ports –Capturing Traffic and Detailed Counters
[admin@guestshell ~]$ sudo tcpdump -i eth1

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
17:04:46.462878 CDPv2, ttl: 180s, Device-ID 'N9K-C93180YC(FDO23010W81)', length 225
17:04:47.301102 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 8014.00:25:46:fa:e5:80.8013,
length 43
17:05:11.973240 DTPv1, length 38
<snip>
17:06:03.073299 ARP, Request who-has 10.102.242.138 tell GW1.cisco.com, length 46
<snip>
75 packets captured [admin@guestshell ~]$ ethtool -S eth1
75 packets received by filter NIC statistics:
0 packets dropped by kernel rx_packets: 719988
tx_packets: 101706
rx_bytes: 60614505
tx_bytes: 11237876
Represents traffic observed rx_broadcast: 31400
only in the CPU/Software path tx_broadcast: 10
rx_multicast: 599041
<snip>
[admin@guestshell ~]$
Bash Shell
Interface State Control via Linux Kernel
• kstack also synchronizes interface up/down state between Linux and NX-OS
• Provides an option for management of the switch in a more Linux/compute friendly
manner via shell directly or scripting. Must be in bash-shell not Guest Shell.
Nexus-9k# run bash
bash-4.2$ ifconfig Eth1-12
Eth1-12 Link encap:Ethernet HWaddr 58:f3:9c:a3:64:dd
inet addr:1.1.1.1 Bcast:1.1.1.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:539 errors:0 dropped:0 overruns:0 frame:0
TX packets:10641 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100
RX bytes:24794 (24.2 KiB) TX bytes:2476821 (2.3 MiB)
bash-4.2$ sudo ifconfig Eth1-12 down
Nexus-9k# show interface ethernet 1/12 | include down

Ethernet1/12 is down (Administratively down)admin state is down, Dedicated Interface
Guest Shell
Run Host CLI and EEM Intergation
[admin@guestshell ~]$ dohost "show version | inc Chassis"
cisco Nexus9000 C93180YC-FX Chassis
[admin@guestshell ~]$ dohost ”config t” “interface eth1/1” “description test”

{ 0 } {}
{ 0 } {}
{ 0 } {}
{ 0 } {}
N9K-C93180YC(config)# feature evmed
event manager applet TEST

event cli match “show version”
action 1 cli guestshell python eem.py
N9K-C93180YC# show version

python: can't open file 'eem.py': [Errno 2] No such file or directory
Use-Case with Guest Shell
In-Situ Packet Generator in Guest Shell
Use spare ports as packet generators
• Guest Shell allows raw socket access to front-panel ports. Thanks to
kstack!
• Open Source packet generator called PackEthCli for CentOS 7 x64 is
available at: http://packeth.sourceforge.net/packeth/Home.html
• PCAP file must be in tcpdump format for this specific tool to work correctly
(PackETH Limitation)
• Send arbitrary L2/L3 traffic out any port or port-channel on N3K/N9K
• Simulate traffic flows for monitoring & troubleshooting
[root@guestshell cli]# ./packETHcli -i Eth1-12 -m 2 -d 10 -n 10000 -f /bootflash/mycap.pcap
Sent 1829 packets on Eth1-12; 124 packet length; 1829 packets/s; 1814 kbit/s data rate;,
2165 kbit/s link utilization
Sent 10000 packets on Eth1-12
In-Situ Packet Generator using Guest Shell
Compiling the packETH binary
• Ensure the Container is sized big enough for the Development Tools and
yum installed.
• Or, build the binary on a VM/Host running CentOS 7 64-bit and copy binary
over
[guestshell@guestshell tools]$ sudo tar -xvf packETH-1.8.tar
[guestshell@guestshell tools]$ cd packETH-1.8/cli/
[guestshell@guestshell cli]$ sudo make all
gcc -g -O2 -Wall -Wunused -Wmissing-prototypes -Wmissing-declarations -c cli_send.ccli_send.c: In
function 'two':cli_send.c:440:42: warning: the omitted middle operand in ?: will always be 'true',
suggest explicit middle operand [-Wparentheses] for(li = 0; pkt2send == 0 ? : li <
pkt2send; li++) { ^cli_send.c:476:41: warning: the omitted
middle operand in ?: will always be 'true', suggest explicit middle operand [-Wparentheses]
for(li = 0; pkt2send == 0 ? : li < pkt2send; li++) { ^gcc -g
-O2 -Wall -Wunused -Wmissing-prototypes -Wmissing-declarations cli_send.o -lm -lpthread -o
packETHcli
[guestshell@guestshell cli]$ ./packETHcli

Usage: packETHcli -i <interface> -m <mode> [-d <delay> -n <number of packets> [-s <startsize
stopsize stepsize] -p period] -f <file>
In-Situ Packet Generator using Guest Shell
Here is the Proof ….
[guestshell@guestshell cli]$ sudo ./packETHcli -i Eth1-12 -m 2 -d 10 -n 5 –f /bootflash/icmp_request.pcap

Sent 5 packets on Eth1-12
[guestshell@guestshell ~]$ sudo tcpdump -i Eth1-12

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on Eth1-12, link-type EN10MB (Ethernet), capture size 65535 bytes
13:11:51.300877 IP guestshell > 1.1.1.2: ICMP echo request, id 756, seq 1, length 64
13:11:51.301269 IP 1.1.1.2 > guestshell: ICMP echo reply, id 756, seq 1, length 64
Thank you
#CLUS
#CLUS

BRKPRG 1816

Uploaded by

Copyright:

Available Formats

You might also like

BRKPRG 1816

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

BRKPRG 1816

Uploaded by

Copyright:

Available Formats

#CLUS

Webex Teams will be moderated cs.co/ciscolivebot#BRKPRG-1816

• Understand how virtualization and containers

with few use-cases

High cost Evolution of Server

machines - model. VMware Full

• Eliminates the need for • Multiple instances of

• Ease of deployment and Cost

Virtualization Cloud Computing

Image vWAAS vSwitch

vCPU vMem vNIC vFirewall vRouter

Curious to know if your Linux

Updating KVM packages

• Containers are light-weight

• Containers shares the host OS and Binary/Lib Binary/Lib

Usually 1 application per container - Microservices

root@server:-$ sudo lxc launch ubuntu:16:04 robot

root@server:-$ sudo lxc list

root@server:~/python/sbfd (kubernetesproject1-197120)$ docker images

root@server:~/python/sbfd (kubernetesproject1-197120)$ docker ps

Kubernetes Masters Cloud Connector

Middleware App Runtime

Monitoring OS, Containers DevOps

Platform-Specific Data Plane

Between DEMAND and CAPABILITY… there is a bridge.

• 3rd Party Application hosting

• Dedicated Filesystem for each

• Ability to install 3rd party applications Party

Cisco Application Hosting Framework

• Unified Interface for application hosting

• Structured Open Network Interfaces

• Rich Deployment Service

• VMAN App Hosting

ISR4321 13000 4GB+ Up to 200 GB VM, LXC 2.7

ISR4331 19000 4GB+ Up to 200 GB VM, LXC 2.7

ISR4451 73000 4GB+ Up to 200 GB VM, LXC 2.7

ASR1K 73000 4GB+ Up to 400 GB VM, LXC 2.7

Catalyst 9K 7400 4GB+ Up to 960GB VM, LXC, Docker* 2.7

* from 16.12 release onwards

C9300# conf t Enable Guest Shell

• VirtualPortGroup interface acts as gateway for Guest 10.1.1.1/24

• Must have a static IP address configured IOS-XE

C9300# guestshell enable Enable Guest Shell

Guest Shell Verification

• Get any base image for the application

• Build the application sudo yum install <>

• Build and prepare VM File system qemu-img create –f vmdk App1

• App VM can be a VMDK file

qemu-img convert –O qcow2 Appl.vmdk App1.qcow2

IOx App Hosting package

• Create a folder and move version: "1.0"

the qcow2 file author-name: "Cisco Systems"

• Create package.yaml with type: vm

the content shown here resources:

IOx App Hosting

115 -rwx 1418270208 May 3 2019 15:12:32 +00:00 App1xe.tar

31963676672 bytes total (26679033856 bytes free)

Install & Active App