BRKIPM 2001 NSF NSR

BRKIPM-2001
v1.1
Routing High Availability NSF & NSR
2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential BRKIPM-2001
4
Agenda
Setting the stage Introduction
Non-Stop Forwarding & Graceful Restart (NSF/GR)
Non-Stop Routing (NSR)
Deployment Considerations and Scenarios
2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
5
BRKIPM-2001
Introduction High Availability
6
Availability Definitions
The probability that an item (or network, etc.) is operational, and
functional as needed, at any point in time
Or, the expected or measured fraction of time the defined service,
device or area is operational; annual uptime is the amount (in
days, hrs., min., etc.) the item is operational in a year
Network Provider
Shared Network
Server
Network
User
Network
Availability
8
Availability Definitions
Network Availability
There is a working network
path between source and
destination (generally bi-
directionally)
Generally involves only the
Network Layer (OSI Layer 3)
Service Availability
The offered service performs
according to the stated SLAs
(packet loss, delay, jitter,
response time, etc.)
Involves all layers
Network vs. Service Availability
Our focus is on Network Availability today
9
What Is High Availability?
DPM = Defects per Million (Hours of Running Time)
Availability Downtime Per Year (24x365)
99.000%
99.500%
99.900%
99.950%
99.990%
99.999%
99.9999%
3 Days
1 Day
53 Minutes
5 Minutes
30 Seconds
15 Hours
19 Hours
8 Hours
4 Hours
36 Minutes
48 Minutes
46 Minutes
23 Minutes
DPM
10000
5000
1000
500
100
10
1
High
Availability
11
Most common causes of downtime
Telco/ISP
35% Human error
31%
Power
failure
14%
Hardware
failure
12%
Other 8%
Common causes of Enterprise Network Downtime **
Embedded Management
Best Practices
System and Network
Level Resiliency
Mitigating the Exposure:
Targeting Downtime
Operational
Process
40%
Network
20%
Software
Application
40%
13
What is Routing High Availability?
Routing HA
Set of technologies & features
to enable traffic to continue to
flow through a device during
a fault
Routing HA maintains the
logical network topology while
the faulty device recovers
Routing HA helps to address
failures within the control
plane of a routing device
Routing HA increases the
resiliency of a single system
14
What is Routing Fast Convergence?
Routing FC
flow around a device during a
fault
Routing FC adapts the logical
network topology to avoid the
faulty component
Routing FC targets to address
any component failure within
a routing device
Routing FC increases the
resiliency of the network
15
What is Routing Fast Convergence?
Routing FC
fault
faulty component
a routing device
16
Routing Convergence vs. Routing HA
Routing FC
fault
faulty component
a routing device
Routing HA
flow through a device during
a fault
Routing HA maintains the
logical network topology while
the faulty device recovers
Routing HA helps to address
failures within the control
plane of a routing device
Routing HA increases the
resiliency of a single system
17
Main Routing HA Applications
Route Processor failure
Routing Process failure
(modular OS)
Chassis Failure
Cat6k-VSS
18
Routing HA to help Planned Downtime
Routing HA technologies can assist minimizing
customer impact during planned maintenance
Controlled RP failover, for example to swap hardware, or to
upgrade memory on RPs
Routing Protocol patches (IOS-XR)
Clearing BGP Sessions (IOS-XR)
HA technologies pre-requisite for In-Service
Software Upgrade
19
BRKIPM-2001
Non-Stop-Forwarding (NSF)
20
Behaviour without NSF
Router A loses its control
plane for some period of time
It will take some time for
Router B to recognize this
failure, and react to it
Control Data
A
Control Data
B
21
Behaviour without NSF
During the time that A has
failed, and B has not detected
the failure, B will continue
forwarding traffic through A
Once the control plane resets,
the data plane will reset as
well, and this traffic will be
dropped
NSF reduces or eliminates the
traffic dropped while As
control plane is down
Control Data
A
Reset
Control Data
B
22
Prerequisite 1: Separated Forwarding Plane
CPU
IOS
interfaces interfaces
Route DRAM
Packet DRAM ASIC NP (Network
Processor)
Interconnect
Control Packet
Data Packet
Data Packet
Control Plane
- RIB (Routing
Information Base)
- aka. routing table
Data Plane
- FIB (Forwarding
Information Base)
Concept of separated control- and forwarding plane
essential for routing HA
Routing HA maintains the forwarding plane while the control
plane restarts/recovers
23
Control
Plane
Engine0 622M
IOS
buff.
SPA
SPA
Q
NP
buff.
Engine5 10G
NP
Q
buff. IOS
IOS
Engine3 3G
Q
F
buff.
F
Q
buff. IOS
Engine6 20G
RP (active) RP (standby)
NP
buff.
NP
Q
buff.
Q
CPU
IOS
CPU
IOS
Data
Plane
Distributed router architectures have this natively
Forwarding information base (FIB) located on Linecards
Cisco 12000
24
IOS IOS
F
IOS IOS
Sup720 (standby)
F
buff.
buff.
buff.
buff.
IOS
F
4, 6, 9, or 13 Linecard/Sup slots
buff.
buff.
SP RP
SP RP
buff.
buff.
buff.
buff.
F
IOS
buff.
buff.
buff.
buff.
buff.
buff.
20G
F
IOS
Catalyst 6500
Cat6500 also has it, despite FIB and Switching Matrix located
physically on RP
FIB is synced between active and standby
25
Prerequisite 2: Stateful Switch Over (SSO)
Any routing HA requires one important mechanism:
The link and its line protocol need to stay up
If not, all neighbours would re-route across the restarting
node
Can be trivial: Keep the linecard up and laser on,
for example for POS/HDLC
Keeping physical link active is easy with Ethernet
as well, but need to sync ARP/v6ND/adjacency
information
Can be complex: PPP, ATM or FrameRelay require
state to be maintained when failing over the control-
plane, sync needed as well
26
GR/NSF Fundamentals
If A is NSF capable, the control
plane will not reset the data
plane when it restart
Instead, the forwarding
information in the data plane is
marked as stale
Any traffic B sends to A will still
be switched based on the last
known forwarding information
This is the Non-Stop
Forwarding behaviour
Control Data
A
No reset
Control Data
B
Mark forwarding
information as stale
27
GR/NSF Fundamentals
While As control plane is
down, the routing protocol hold
timer on B counts down....
A has to come back up and
signal B before Bs hold timer
expires, or B will route around
it
When A comes back up, it
signals B that it is still
forwarding traffic, and would
like to resync
This is the first step in
Graceful Restart (GR)
Hold Timer: 15 14 13 12 11 10 9876
Control Data
A
Control Data
B
28
GR/NSF Fundamentals
The second GR phase deals
with neighbors updating the
restarting routers routing table
This involves new protocol
mechanisms
Control Data
Control Data
A
B
I
m
r
e
s
t
a
r
t
i
n
g
O
k
,

f
i
n
e
,

I
l
l

s
e
n
d

r
o
u
t
e
s
29
GR/NSF Fundamentals Summary
Key Components of NSF on the restarting
router
Keeping interfaces/linecards up
Maintaining Forwarding State in the data plane
Synchronizing routing information post failover
On the neighbouring router(s)
Maintain routes while neighbour restarts
Help restarting node synchronizing its routing table
GR/NSF implementation in various protocols
generally differ in the way synchronization
works
NSF/GR
capable
NSF/GR
aware
30
EIGRP GR/NSF Fundamentals
The signal in EIGRP is an
update with the initialization
and restart (RS) bits set.
A sends its hellos with the
restart bit set until GR is
complete.
B transmits the routing
information it knows to A.
When B is finished sending
information, it sends a special
end of table signal so A knows
the table is complete
A
B
T
o
p
o
l
o
g
y

i
n
f
o
r
m
a
t
i
o
n
H
e
l
l
o

+

R
e
s
t
a
r
t
I
n
i
t

+

R
e
s
t
a
r
t
E
n
d

o
f

t
a
b
l
e
Control Data
Control Data
31
Control Data
EIGRP GR/NSF Fundamentals
When A receives this end of
table marker, it recalculates its
topology table, and updates
the local routing table
When the local routing table is
completely updated, EIGRP
notifies CEF
CEF then updates the
forwarding tables, and
removes all information
marked as stale
A
B
Control Data
32
EIGRP GR/NSF Configuration
Use the nsf command under
the router eigrp configuration
mode to enable graceful
restart
no configuration required on
helper node
Show ip protocols can be
used to verify graceful restart
is operational
Currently only supported for
IPv4
A
B
router eigrp 100
nsf
....
A#show ip protocols
Routing Protocol is "eigrp 100
....
Redistributing: eigrp 100
EIGRP NSF-aware route hold timer is 240s
EIGRP NSF enabled
NSF signal timer is 20s
NSF converge timer is
....
http://www.cisco.com/en/US/tech/tk365/technologies_white_paper0900aecd8023df74.shtml
http://www.cisco.com/en/US/products/sw/iosswrel/ps1839/products_feature_guide09186a0080160010.html
Restarting Node
Helper Node
33
OSPF NSF Implementations
There are two mechanisms: Cisco- and IETF
(RFC3623) Style
cisco-Style is also defined as in informational RFC4811 &
RFC4812
Approaches differ in the ways ...
the restart process is signalled
the restarting node synchronizes the LSA database
deciding when to abort the GR process
34
OSPF NSF Cisco Style
OSPF uses an extension to
the hello packets called link
local signaling
The first hello A sends to B
has an empty neighbor list;
this tells B that something is
wrong with the neighbor
relationship
A sets the restart bit in its
hello, which tells B that A is
still forwarding traffic, and
would like to resynchronize its
database
A
B
E
m
p
t
y

H
e
l
l
o

+

R
e
s
t
a
r
t
Control Data
Control Data
35
B moves A into the exchange
state, and uses out of band
signaling (OOB) to
resynchronize their databases
This process is the same as
initial database
synchronization, but it uses
different packet types
A
B
D
B
D

e
x
c
h
a
n
g
e
Set A to
exchange
L
S
A

e
x
c
h
a
n
g
e
Control Data
Control Data
36
Control Data
When A and B have
resynchronized their
databases, they place each
other in full state, and run SPF
After running SPF, the local
routing table is updated, and
OSPF notifies CEF
marked as stale
A
B
Control Data
37
OSPF NSF CISCO Configuration
A
B
router ospf 1
nsf cisco
router ospf 1
Restarting Node
Helper Node
B#show ip ospf int
GigabitEthernet0/0 is up, line protocol is up
Supports Link-local Signaling (LLS)

Cisco NSF helper support enabled
IETF NSF helper support enabled
A#show ip ospf
Non-Stop Forwarding enabled

IETF NSF helper support enabled
Cisco NSF helper support enabled
A#show ip ospf neighbor det
Neighbor 10.0.0.3, interface address 10.0.2.34
In the area 0 via interface GigabitEthernet4/1
Neighbor priority is 1, State is FULL, 6 state changes
DR is 10.0.2.34 BDR is 10.0.2.33
Options is 0x12 in Hello (E-bit, L-bit)
Options is 0x52 in DBD (E-bit, L-bit, O-bit)
LLS Options is 0x1 (LR)
38
OSPF NSF IETF Style
OSPF IETF NSF uses a new
LSA type to signal GR
A will send out a GRACE-LSA
to inform its neighbour(s) that
it is undergoing a graceful
restart
A
B
G
r
a
c
e

L
S
A
Control Data
Control Data
39
OSPF NSF IETF Style
B moves A into the exchange
state, and uses the regular
mechanism to resynchronize
their databases
This process is the same as
initial database
synchronization
A
B
D
B
D

e
x
c
h
a
n
g
e
Set A to
exchange
L
S
A

e
x
c
h
a
n
g
e
Control Data
Control Data
40
Control Data
OSPF NSF IETF Style
When A and B have
databases, they place each
other in full state, and run SPF
OSPF notifies CEF
marked as stale
(all of the above is identical to
OSPF NSF style)
A
B
Control Data
41
OSPF NSF IETF in Operation
A
B
Restarting Node
Helper Node
A# redundancy force-switchover
This will reload the active unit and force switchover to standby[confirm]y
Preparing for switchover..
#B
*Dec 15 10:13:40.374: OSPF: IETF NSF Received grace-LSA from 10.0.0.2 on GigabitEthernet4/1
*Dec 15 10:13:40.374: OSPF: IETF NSF Validate grace-LSA from nbr 10.0.0.2 on GigabitEthernet4/1
*Dec 15 10:13:40.374: OSPF: IETF NSF Process grace-LSA from nbr 10.0.0.2 on GigabitEthernet4/1,
age 1, grace period 120, graceful restart reason: Switch to redundant control processor,
graceful ip address 10.0.2.33
*Dec 15 10:13:40.374: OSPF: IETF NSF helper interface count: 1 (area 0), GigabitEthernet4/1
*Dec 15 10:13:40.374: OSPF: IETF NSF Enter graceful restart helper mode for 10.0.0.2 on
GigabitEthernet4/1 for 119 seconds (requested 120 sec)
*Dec 15 10:14:04.266: OSPF: IETF NSF GR-resync FROM Nbr 10.0.0.2 10.0.2.33 GigabitEthernet4/1
*Dec 15 10:14:04.266: OSPF: IETF NSF Starting graceful-resync with 10.0.0.2 address 10.0.2.33 on
GigabitEthernet4/1
*Dec 15 10:14:04.266: %OSPF-5-ADJCHG: Process 1, Nbr 10.0.0.2 on GigabitEthernet4/1 from LOADING
to FULL, Loading Done
42
OSPF IETF/RFC3623 vs. Cisco
Main practical difference is in criteria for aborting the GR
process
RFC3623 aborts the process when
it detects a neighbour which is not OSPF-GR aware, or
if a topology change occurs during the LSDB synchronization
Cisco NSF continues the process, accepting the caveat
of transient routing asymmetry
nsf cisco enforce global can be used to abort NSF when
non-GR-aware neighbors are found
I feel the nsf cisco being more flexible, at the expense
of being proprietary
You need to settle on one mode, however any Cisco box
supporting both modes can help a neighbour configured
with any of the two while the neighbour restarts
43
ISIS NSF Implementations
For ISIS, there are also two approaches: nsf cisco
and nsf ietf
Unlike OSPF, approaches differ more
fundamentally:
IETF/RFC3847 works more like a traditional GR/NSF
protocol
cisco-style ISIS NSF does not require any protocol
extensions or neighbour awareness
44
IS-IS GR/NSF Fundamentals (IETF)
IS-IS adds a new TLV to the
hello packet, the restart option.
The restart option TLV
contains a Restart Request
(RR) bit and a Restart
Acknowledgement (RA) bit
Restart option TLV needs to
be sent in all hellos (IIH).
When A restarts, it transmits
its hellos with an empty
neighbor list, and the RR bit
set
B transmits hellos to A with the
RA bit set
A
B
E
m
p
t
y

H
e
l
l
o

+

R
R
H
e
l
l
o

+

R
A
Control Data
Control Data
45
B then clears the flags which
indicate routing data that
needs to be transmitted to A
(the SRM flags)
A and B then use IS-IS normal
synchronization process using
complete sequence number
packets (CSNPs) to describe
their databases, and
exchanging link state packets
(LSPs)
A
B
C
S
N
P
s
L
i
n
k

S
t
a
t
e

P
a
c
k
e
t
s
clear SRM flags
Control Data
Control Data
46
Control Data
When A and B have
databases, they run SPF
IS-IS notifies CEF
marked as stale
A
B
Control Data
47
IS-IS GR/NSF IETF Configuration
Use the nsf ietf command
under the router isis configuration
mode to enable graceful restart
No configuration required on helper
node (enabled by default)
show isis nsf can be used to
verify graceful restart is operational
show clns neigh detail
shows neighbor support of ISIS GR
A
B
router isis
nsf ietf
....
A#show isis nsf
NSF is ENABLED, mode ietf'
A#show clns neighbor detail

System Id Interface SNPA State
neighborxx Gi7/1 0005.0096.a819 Up Area
NSF capable
Restarting Node
Helper Node
48
IS-IS GR/NSF Fundamentals (Cisco-Style)
IS-IS Cisco-Style works
without any GR protocol
extensions
IS-IS constantly syncs the
neighbour adjacency state as
well as LSP header
checkpoints on the standby
Once A restarts, it requests the
full LSPs from its neighbors,
using a CSNP (Complete
Sequence Number Packet)
packet
Neighbour follows regular IS-
IS mechanisms and floods its
complete LSP database
A
B
C
S
N
P
L
S
P
s
Control Data
Control Data
49
Control Data
Control Data
IS-IS GR/NSF Fundamentals (Cisco)
When A has resynchronized its
database, A runs SPF
IS-IS notifies CEF
marked as stale
A
B
50
IS-IS GR/NSF Cisco Configuration
Use the nsf cisco command
under the router isis configuration
mode to enable graceful restart
No configuration required on
helper node
A
B
router isis
nsf cisco
....
A#show isis nsf
NSF is ENABLED, mode cisco'
Restarting Node
Helper Node
51
IS-IS IETF/RFC3847 vs. Cisco
With nsf cisco requiring no
protocol extensions to
synchronize the LSDB,
deploying it is much easier
Cisco nodes configured with
nsf cisco will also signal
support for neighbours using
IETF-style GR
A
B
router isis
nsf cisco
....
B#show clns neighbor detail
System Id Interface SNPA
neighborxx Gi4/3 0005.00fe.3444
NSF capable
router isis
nsf ietf
....
52
BGP GR/NSF Fundamentals
Graceful restart capability is
negotiated when session comes
up. If both peers state they are
capable of GR, its enabled on the
peering session, on a per-
address-family (ipv4, ipv6, vpnv4,
etc.) basis
When A restarts, it opens a new
TCP session to B, using the same
router ID
B interprets this as a restart, and
closes the old TCP session
B also considers TCP session going
down as a signal for A restarting
While A restarts, B marks all paths
received from A as stale
A
B
G
R

c
a
p
a
b
i
l
i
t
y
N
e
w

T
C
P

S
e
s
s
i
o
n
Restart; close
old session
r3#show ip bgp 10.20.0.0
BGP routing table entry for 10.20.0.0/16, version 47
Paths: (1 available, best #1, table Default-IP-Routing
Flag: 0x820
Not advertised to any peer
Local, (stale)
10.0.0.2 (metric 21) from 10.0.0.1 (0.0.0.0)
Origin IGP, metric 0, localpref 100, valid, internal, best
Control Data
Control Data
53
B transmits updates containing
its BGP table (its local RIB
out)
A goes into read only mode,
and does not run the bestpath
calculations until its B has
finished sending updates
When B has finished sending
updates, it sends an end of
RIB marker, which is an
update with an empty
withdrawn NLRI TLV
A
B
U
p
d
a
t
e
s
E
n
d

o
f

R
I
B

M
a
r
k
e
r
Read only
mode
Control Data
Control Data
54
Control Data
When A receives the end of
RIB marker, it runs bestpath,
and installs the best routes in
the routing table
After the local routing table is
updated, BGP notifies CEF
marked as stale
A
B
Control Data
55
Use the bgp graceful-restart
command under the global router bgp
configuration mode to enable graceful
restart
IOS-XR and recent IOS can
disable it on a per-nbr basis
Needs to be enabled on both ends,
sessions need to be reset in order for
the config to take effect
Show ip bgp neighbors can be
used to verify graceful restart is
operational
A
B
router#show ip bgp neighbors x.x.x.x
....
Neighbor capabilities:
....
Graceful Restart Capabilty:advertised and received
Remote Restart timer is 120 seconds
Address families preserved by peer:
IPv4 Unicast, IPv4 Multicast
router bgp 65000
bgp graceful-restart
....
router bgp 65501
....
56
GR/NSF Summary
All NSF protocols require some form of neighbour
interaction and functionality/configuration on the
adjacent systems
Holding onto the routes while the neighbour restarts
Re-Sending the routing information
Deploying NSF in scaled edge deployments (for
example large hub site or service provider edge)
can be challenging as all neighbors need to be
touched (config, OS upgrade, etc.)
What if we used another approach
57
BRKIPM-2001
Non-Stop Routing
58
Non-stop Routing NSR
Idea: Why not sync all
routing protocol state to
the standby RP (or
standby process)?
Restarting RP could pick
up right where the primary
left off
No need to refresh any
information, no need for
the neighbour to know that
anything happened
Easy idea challenging
implementation
Now we absolutely need to
avoid anything to let the
neighbour know
Forwarding
Continues
A
c
t
i
v
e
S
t
a
n
d
b
y
SSO
Line Cards
Routing
Adjacency
Maintained to
Neighbours
No Link Flap
59
The easy NSR
IS-IS nsf cisco (available for a long time) actually looks
like NSR (only on the surface, though)
Checkpointed adjacency state (as maintained by hellos) as
well as LSDB on standby, able to recover with existing
protocol mechanism
Neighbour actually notices something happens, but we still
achieve non-stop forwarding
RSVP and PIM in IOS-XR uses checkpoints, refreshes
state from neighbors
There is a substantial difference, to real NSR, though:
restarting node forwards on potentially outdated
information
Lets look at real NSR now
60
OSPFv2 NSR (IOS-XR)
Neighbour & interface state
and LSDB constantly
synced between active and
standby
Input packets replicated to
both active and standby (1)
LSDB updated on active &
standby (2a/2b)
Standby ACKs LSA to
Active (3)
Active RP acks LSA to
sender (4)
state & LSDB sync
(4)
(3)
(1)
ACTIVE RP
OSPF
Raw IP
(2a)
OSPF
Raw IP
(2b)
STANDBY RP
Sender/Peer
61
More tricky: NSR for TCP-based Protocols
LDP and BGP use TCP for reliable delivery of
PDUs
Eases protocol implementation, but makes NSR quite
challenging
Strict requirement to maintain TCP session during
failover
TCP session reset would be interpreted by nbr as adjacency
down rerouting
How can we reliably maintain the TCP session?
Need to ensure TCP stack on active and standby RP are
synced (sequence numbers, etc.)
Need to ensure to only acknowledge the receipt of a packet
when primary and standby received it
62
TCP NSR Receive Path (IOS-XR)
Input pkt replicated to
both active and standby
TCP stack (1)
Standby ACKs pkt to
active once it stored it in
buffer (2)
Once active TCP sees
the ACK, it ACKs pkt to
sender
Active owns TCP
session
TCP delivers data to
application
(4)
(2)
(1)
ACTIVE RP
APP
TCP
(4a)
APP
TCP
(4b)
STANDBY RP
(3)
Sender/Peer
63
TCP NSR Send Path (IOS-XR)
In the send path, standby
TCP stack sends the
packet towards the peer
Standby owns the
session
(4)
(2)
ACTIVE RP
APP
TCP
(1)
APP
TCP
STANDBY RP
(3)
Sender/Peer
64
NSR Support in IOS-XR
Supported for BGP, OSPFv2,
and LDP
OSPFv3/IPv6 planned for 4.2
Configured on global protocol
level
When GR/NSF is also enabled,
protocols can fall back to NSF in
case NSR is not possible
for example when standby RP is
not in NSR-ready state
generally recommended to enable
NSF alongside NSR
Important to monitor NSR state
on standby
router bgp
nsr
router ospf ..
nsr
mpls ldp
nsr
router isis
nsf cisco
RP/0/RP0/CPU0:router#show redundancy
Redundancy information for node 0/RP0/CPU0:
==========================================
Node 0/RP0/CPU0 is in ACTIVE role
Partner node (0/RP1/CPU0) is in STANDBY role
Standby node in 0/RP1/CPU0 is ready
Standby node in 0/RP1/CPU0 is NSR-ready
65
NSR Support in IOS
BGP NSR
Supported for IPv4 VRF
neighbors on c10k and c7600
GR/NSF should also be
enabled
For peers supporting GR, TCP
state is not maintained and
failover is done via NSF
OSPFv2 NSR
coming in 15.1(2)S
GR/NSF can be enabled to
support fallback to NSF in case
NSR not ready
router bgp
address-family ipv4 vrf ..
neighbor x.x.x.x ha-mode sso
....
# show ip bgp vpnv4 all sso summary
# show tcp ha connections
router ospf 1
nsr
[ nsf cisco|ietf ]
....
# show ip ospf nsr
66
NSR Summary
Unique, Self-Contained Routing HA Solution
Simplifies NSF/SSO deployment by synchronizing
edge routes automatically
NSF-aware neighbour devices not needed
Addresses additional network scenarios e.g. unmanaged
CPE devices
Delivers persistent routing for the entire customer
edge
Retains scalability and safety of NSF/GR with
benefits of NSR
67
BRKIPM-2001
Deployment Considerations and Use
Cases
68
Complex?!?!
Two approaches (NSF and NSR) to address the
same problem
Different protocols, different NSF/NSR variants,
implementations and roadmaps
Different fundamental approaches to increase
availability: HA and Fast Convergence
Lets look at some generic deployment guidance,
some implementation caveats and use cases
69
GR/NSF Deployment Considerations
Be careful with partial
deployments of GR/NSF
capability
If B restarts, A will reset its
session, removing all the
routing information it learned
from B. However, D will
continue to forward traffic
through B
This will, at best, cause
asymmetric routing. At worst, it
could cause a routing loop
Router A must be GR capable
or GR aware
Core
GR/NSF capable
A
B C
D
Session reset
D continues
forwarding
Asymmetric
return path
70
Service Provider
A
B C
D
OSPF
Multiple Routing Protocols
OSPF is configured for
GR/NSF, while BGP is not
Ds next hop for all routes is A;
the path to A is learned via
OSPF
If the control plane on B
restarts, D will continue
learning BGP routes from C
with a next hop of A; it will also
maintain the best path to that
next hop through B
Best path
to A
BGP learned
routes
71
Multiple Routing Protocols
Since the best path to A is still
through B, D will continue
forwarding through B for all the
BGP routes it is learning
through C
B will drop this traffic, since it
is not maintaining its BGP
state, only its OSPF state
If BGP and an IGP are running
together, they must both have
GR enabled
Service Provider
A
B C
D
OSPF
D continues
forwarding
BGP learned
routes
72
A
B C
D
IPv6 Deployment Considerations
NSF/NSR implementation for
IPv6 is not yet at the same
state as for IPv4, i.e.
no GR support for IPv6-AF in BGP
in IOS
no NSF support for OSPFv3
but: works with IS-IS
As v4 and v6 routing is
carried in different protocols,
everything is fine
IPv4
continues
through
restarting
node
IPv6
routes
around the
failure
73
MPLS Deployments P/LSR Routers
MPLS P (or LSR) routers act as transit
node only
no directly connected customers or
services
Assuming there is sufficient
redundancy and capacity within the
network, it can be better just route
around the failure
There are still several deployments
around with IOS releases not
supporting MPLS SSO
RPR redundancy should be configured
to let linecards reload on RP
failure/failover
Fast Convergence required to minimize
packet loss
A
B C
D
74
Other Protocols
To achieve hitless convergence, all protocols and
features involved in routing and forwarding of a
given service along a given path need to be GR
enabled- or capable
All routing protocols
Dont forget PIM (Mcast), RSVP (MPLS-TE)
ARP/IPv6 ND
HSRP/VRRP
etc.
Did we miss anything?
75
HA with NAT/FW/IPSec/L2TP
Network Address Tranlsation (NAT), Firewall or
IPSec/L2TP/PPPoX all maintain session state
Broadband platforms (ASR1000, c10k, ASR9000)
support SSO for PPPoX/L2TP to allow for stateful
switch-over
ASR9000 maintains session state on linecard(s), so state is
much easier to maintain for RP failovers
Currently, IPSec (incl. DMVPN), NAT and FW is not
SSO- capable on any platform, so sessions need to be
re-established after RP failover
Lack of SSO support for a fundamental feature like the
ones above on a given platform is often a reason to not
deploy Routing HA at all
We rather want to fail over to a redundant device
76
Protocol Hello Considerations
Depending on platform and OS, it can
take a few seconds until standby
process is operational
Neighbour adjacencies configured with
fast hellos could time out, leading to re-
route
Default hello timers are ok, no need to
increase
Restarting RP/process starts to send
hellos as soon as possible and at higher
rate right after restart
Make sure to test failover with tuned
hello times with platforms/software prior
to deployment (see [1] for some test
results)
[1] http://www.cisco.com/en/US/technologies/tk869/tk769/technologies_white_paper09186a00801dce40.html
%OSPF-5-ADJCHG: [],
Neighbor Down: Dead timer
expired
77
BFD Consideration
BFD (Bi-directional Forwarding Detection)
is a hello-type protocol designed and
deployed to provide sub-second failure
detection
BFD needs to be SSO-aware to ensure standby RP can take
over
BFD session state synced
Still, platform restrictions apply, ex. 6500/7600 performing RP
failover cause short traffic disruption on bus, affecting traffic
to/from the RPs
S/E chassis and 67xx/ES linecards mitigate this
Still: recommended not to go below 500msec x 3, smaller values
can cause BFD going down
BFD
BFD
OSPF OSPF
78
Single-RP Deployments
Any platform only
supporting a single
control plane (i.e. 7200,
ISRs, fixed Catalyst L3-
switches, etc.) can only
act as GR helper node
SSO and NSF is not
configurable
When BGP GR is
configured to act as
helper, they wont
announce GR for any
address family (AF)
10.0.0.2
7600,
dual RP
7200
router#show ip bgp neighbors 10.0.0.2
....
...
Graceful Restart Capability: advertised and received
Address families advertised by peer:
none
....
...
IPv4 Unicast
10.0.0.1
79
Single-RP Deployments
Problematic are dual-RP platforms (i.e.
6500, 7600, 12000, ASR1000) with only a
single RP installed
In this case, redundancy mode can be
configured as RPR, documenting that
linecards/etc. will be restarted when RP
reloads
NSF should not be configured for any protocol,
helper support is generally enabled by default
However, configuring BGP GR (to act as
helper) will announce GR for supported/
configured AFs
Neighbors will hold on to routes if peer goes
down
Recommendation: Avoid single-RP
deployments when using NSF/GR
7600
single RP
....
...
IPv4 Unicast
80
Example, using multiple AFs
Remote node shutdown, no failover
router#show bgp all neighbors 10.0.0.1 routes
Network Next Hop Metric LocPrf Weight Path

*>i10.20.0.0/16 10.0.0.2 0 100 0 I

*>i2001:DB8:200::/56 2001:DB8:1::2 0 100 0 I
*Nov 24 14:31:55.487: %BGP-5-ADJCHANGE: neighbor 10.0.0.1 Down NSF peer closed the session
*Nov 24 14:31:55.487: IPv6RT[Default]: bgp 65000, Delete 2001:DB8:200::/56 from table
router#show bgp all neighbors 10.0.0.1 routes
For address family: IPv4 Unicast
BGP table version is 43, local router ID is 10.0.0.3
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete
S>i10.20.0.0/16 10.0.0.2 0 100 0 I
router#
*Nov 24 14:33:55.323: RT: del 10.20.0.0/16 via 10.0.0.2, bgp metric [200/0]
*Nov 24 14:33:55.323: RT: delete subnet route to 10.20.0.0/16
router#
Routes not purged until the GR stale timer expires (2 mins by default)
no support for IPv6
GR in the test setup
81
BGP GR and Manual Session Resets
Helper node (B) considers TCP session
reset as an indication for A restarting
B holds on to the routes via A
If A reloads or operator on A clears the
session, we would rather B to purge the
routes and converge around A
BGP supports the CEASE notification: B
would interpret this as a real reset and
route around
Caveat: IOS currently does not send CEASE
prior to reload, nbr shutdown or when doing
clear bgp
IOS-XR and NX-OS send CEASE notification as per
RFC 4486
No compelling workaround is available,
were working on getting this implemented in
IOS
A
B
Restarting Node
Helper Node
TCP
Session
82
Fast Convergence
Sub-second IGP convergence can
generally be achieved, thanks to feature
development and implementation
improvements in the past few years
Rapid failure detection often the biggest
challenge
Robust implementation, mitigating the
risk of churn
Thanks to BGP Prefix-Independent
Convergence (BGP-PIC), even very
large BGP tables can converge as
quickly as the underlying IGP
Fast Convergence technologies enable
IP networks to offer strict SLAs for
mission-critical, loss-sensitive
applications
83
Interaction with Fast Convergence
Which failure types can be addressed by HA and by
Routing Convergence?
Failure Routing HA Routing FC
Link Failure No Yes
Node Failure No Yes
Process Failure Yes No *
RP Failure/
Failover
Yes Yes **
*) Some process failures result in effective re-routing, others could lead to blackholes
**) Detection of RP failover depends on HA config
84
Design approach for Fast Convergence
Deploy redundant devices/links to provide
path diversity for any single failure case
Detect failures as fast as possible and
route around
Send notification to other devices so they
can also route around
Fast Convergence addresses both link
and node/RP failure, while routing HA
only addresses RP/protocol failover
Link failures are more common than
node/RP failures, hence we need to
look at Fast Convergence to address
those anyway
Why not just rely on Fast Convergence for
node/RP failures?
85
Fast Convergence generally works
extremely well to route around RP
failures in the core and distribution, i.e.
within the core IGP domain
Core generally designed with enough
capacity to allow for single device/link
failures
Same level of convergence often cant be
delivered into the access
Distribution routers are therefore a sweet
spot for Routing HA
Failure has impact to a large number of
customers
Can provide lossless failover for PE RP
failures
Can minimize downtime for software
upgrades
Core
Dist.
Access
86
Deploying HA in the Distribution
Option 1: NSF disabled in core
Core
Dist.
Access
NSF and SSO disabled in Core devices,
and enabled within distribution layer
Core routers:
OSPF/ISIS NSF helper-mode only to support distribution routers
BGP & LDP dont have helper-mode, need to enable GR to support
distribution nodes
Core router RP failure will trigger routing convergence, use of LDP
labels or BGP paths follows IGP
Distribution routers:
Dual-RP Nodes: All protocols enabled for graceful restart
Single-RP or non-redundant nodes: no NSF/SSO/GR-helper config
No problem, core and access neighbors will route around (if possible)
Enable NSR when available (or ISIS nsf cisco)
Access Routers:
GR helper mode enabled, where available
If IGP is run into the access layer, ensure all access routers are
GR-aware for IGP, otherwise use OSPF nsf cisco on distribution
routers, which doesnt abort GR if some neighbors are not GR-
aware
87
Deploying HA in the Distribution
Option 2: NSF in Core and Dist.
NSF and SSO enabled on Core as well as
Distribution devices
Core routers:
Imperative to enable NSF/GR for all protocols,
incl. IGP, BGP, LDP, RSVP, etc.
Distribution routers:
Dual-RP Nodes: All protocols enabled for graceful restart, and NSR
(when available)
Single-RP: needs GR-helper config to support core router failover
Redundant nodes with single RP: Problematic, can cause black hole
Access Routers:
GR helper mode enabled, where available
If IGP is run into the access layer, ensure all access routers are GR-
aware for IGP, otherwise use OSPF nsf cisco on distribution routers,
which doesnt abort GR if some neighbors are not GR-aware
Core
Dist.
Access
88
NSF/GR Deployment Considerations
Summary
Evaluate NSF/GR/SSO support for all relevant
protocols and features
Check for non-standard hello and very low BFD
timers
Be aware of single-RP deployments and its
dependencies on NSF/GR (especially BGP)
When core provides enough capacity to re-route
around failures, consider NSF/GR in distribution
only
Remember that NSF/GR only addresses selected
failure scenarios, ensure routing convergence is
tuned to handle link and node failures quickly
89
NSR Deployment Considerations
Restarting node doesnt need any neighbour awareness,
considerations with regards to neighbour capabilities
doesnt really apply
Partial deployments on selected routers easily possible
What still applies:
All protocols/features need to be HA/NSR- and SSO-
capable
In addition, we generally recommend enabling NSF as a
fallback to NSR Restarting router reverts to NSF in
case NSR recovery failed (or NSR wasnt ready/synced
at time of failure)
Hence: Unless pure NSR deployment is targeted, same
considerations/evaluations apply
90
NSR IOS-XR Case Study ASR9000
ASR9000 MPLS-VPN P & PE Device
ISIS, configured with nsf ietf
LDP: NSR & GR
BGP: NSR & GR (L3VPN, VPLS AD)
Multicast
Test Results
RP Failover without HA: 30-140 sec
traffic loss
RP FO, RP removal with HA: 0 ms
(vpnv4 and VPLS flows)
Link failures (core and edge links): 140-
300 msec
nsr process-failures switchover
router isis FOO
nsf ietf
address-family ipv4 unicast
spf-interval initial 100 sec 100 max 1000
interface
bfd fast-detect ipv4
bfd minimum-interval 50
bfd multiplier 3
mpls ldp
nsr
graceful-restart
log graceful-restart
router bgp
nsr
bgo graceful-restart
bgp graceful-restart graceful-reset
multicast-routing
address-family ipv4
nsf
91
Use Case: Cat6500 VSS
Catalyst VSS is a special form of RP
redundancy
Active RP is in one chassis, hot-
standby RP in the other
State synchronisation/SSO achieved
via VSL
RP or chassis fail-over requires
Routing HA mechanisms (NSF) in the
same way as in a single, dual-RP
chassis
Current IOS SW releases offer VSS
NSF/SSO feature parity compared to
single chassis HA deployments
Using Quad-Sup deployment doesnt
change this, redundant Sup in the
chassis is not synced to active, Sup
failure will trigger chassis reload
Si Si Si Si
Physical View
Logical View
VSL
Active
Standby
Active Standby
92
Use Case: IOS Software Redundancy on
Single-RP ASR1000
Stand-by IOS process in
RP in the single-engine
4RU/2RU system
Two IOS process in a
single RP function similar
to different processes on
separate RP
Supports all NSF/SSO
features supported by
dual-RP systems
Requires additional RP
memory 4G
Route Processor
Linux Kernel
IOS
Backup
Chassis
Manager
Interface
Manager
Forwarding
Manager
IOS-XE Middleware
IOS
Active
93
Use Case: In-service Software Upgrade (ISSU)
STANDBY ACTIVE
OLD NEW
= RP Is Active = RP Is Standby = New Cisco IOS = Old Cisco IOS
1
2
3 4
5
OLD
ACTIVE
OLD
STANDBY
OLD
ACTIVE
NEW
STANDBY
OLD
STANDBY
NEW
ACTIVE
OLD
STANDBY
NEW
ACTIVE
NEW
STANDBY
NEW
ACTIVE
94
BRKIPM-2001
Summary
95
Routing HA Evolution
Data Plane
Control Plane
Data Plane
Control Plane
Data Plane
Control Plane
Data Plane
Control Plane
Data Plane
Control Plane
Data Plane
Control Plane
None
NSF
NSR
Failure
Propagation
Restarting Node Neighbor
96
Key Takeaways
NSF & NSR technologies augment the portfolio of
technologies to increase network availability, offering
(near-)zero packet loss for control plane failures
Designing for Fast Routing Convergence has been #1
priority in most networks and has proven to be very
successful
Complexity of NSF/GR deployment often made it 2
nd
choice, treated with lower urgency
Introduction of NSR changes the game, really eases
deployment as it acts locally per node
ISSU requires Routing HA to reduce downtime
Its time to look at HA again!
97
Cisco Nonstop Forwarding with Stateful Switchover Deployment
Guide
http://www.cisco.com/en/US/technologies/tk869/tk769/technologies_white_paper
0900aecd801dc5e2_ps6550_Products_White_Paper.html
Cisco Globally Resilient IP: Overview and Applications
http://www.cisco.com/en/US/docs/ios/solutions_docs/grip/GRIP_ovr.html
Please also browse the on-site Cisco Store for suitable reading
BRKIPM-2001
Recommended Reading
98
We value your feedback - don't forget to complete your online
session evaluations after each session. Complete 4 session
evaluations & the Overall Conference Evaluation (available from
Thursday) to receive your Cisco Networkers 20
th
Anniversary t-shirt.
All surveys can be found on our onsite portal and mobile website:
www.ciscoliveeurope.com/connect/mobi/login.ww
You can also access our mobile site and
complete your evaluation from your mobile
phone:
1. Scan the Access Code
(See http://tinyurl.com/qrmelist for software,
alternatively type in the access URL)
2. Login
3. Complete and Submit the evaluation
Please complete your Session Survey

BRKIPM 2001 NSF NSR

Uploaded by

Copyright:

Available Formats

You might also like

BRKIPM 2001 NSF NSR

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

BRKIPM 2001 NSF NSR

Uploaded by

Copyright:

Available Formats

BRKIPM-2001

Supports Link-local Signaling (LLS)

Non-Stop Forwarding enabled

A#show clns neighbor detail

Network Next Hop Metric LocPrf Weight Path

Network Next Hop Metric LocPrf Weight Path

You might also like