uCPE[4]- uCPE acting as a Centralized INET Gateway for Remote SD-WAN Branches

Parbhat Kapoor

parbhat@versa-networks.com

1 © 2017 Versa and/or its affiliates. All rights reserved. Versa Networks Confidential
Purpose:

1. This document will shed a light on how to make uCPE act as an Internet Gateway for Remote SDWAN branches.

2. We will also see that with default configuration on both FlexVNF & Palo Alto Remote-SDWAN Branch won’t be able to
communicate with Internet destinations and how we mitigate from that situation.

2 © 2017 Versa and/or its affiliates. All rights reserved. Versa Networks Confidential
Scenario 1a: Traffic coming from Remote SDWAN Branch towards Internet is getting blocked by Palo Alto
1. Remote SDWAN was unable to reach Internet destinaton but was able to reach uCPE Local LAN-VR Subnet 172.16.191.0/24 with below default configuration on:

a) uCPE was configured with DIA + Gateway capability.

b) Palo Alto had mainly 2 static routes which were suffice to communicate between Remote SDWAN branches and uCPE local LAN-VR subnet.
1) Default route pointing towards Untrust interface (towards Internet/Remote SD-WAN destination)
2) Static route pointing towards Trust interface in order to reach LAN-VR subnet (towards uCPE LOCAL LAN-VR subnet)

Above configuration is not suffice for Remote SD-WAN to start using uCPE as a Centralized Firewall INET/Edge Gateway

3 © 2017 Versa and/or its affiliates. All rights reserved. Versa Networks Confidential
parbhat@parbhat-Standard-PC-i440FX-PIIX-1996:~$ traceroute 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets MPLS 8.8.8.8
Server
1 *** INTERNET
172.16.125.90
2 ***
Remote SDWAN Branch WAN IP: 192.168.2.1/24
3 ***
4 192.168.2.6 (192.168.2.6) 87.865 ms 87.974 ms 87.945 ms – INET-VR WAN IP: 192.168.2.6/24
5 ***
Vni0/0.0
6 ***
… Internet-Transport-VR
17 * * *
parbhat@parbhat-Standard-PC-i440FX-PIIX-1996:~$ Return traffic
tvi0/602.0 W-ST-Marriott-LAN-VR-PK-INET
Internet Destined traffic

dtvi-0/41 Versa FlexVNF Firewall

tvi0/11.0
Palo Alto Firewall
DIA TRAFFIC PATH
Marriott-Control-VR/MP-BGP/Tunnels

mpls-vpn-core-instance

tvi0/603.0 L-ST-Marriott-LAN-VR-PK-INET

Palo Alto VM

172.16.20.2 Untrust
VRF: Marriott-LAN-VR Vni-0/303.0 172.16.20.1

Vni-0/302.0 172.16.10.1 172.16.10.2 Trust

Lan: 172.16.191.1/24 Vni0/1.0

172.16.191.2/24

4 © 2017 Versa and/or its affiliates. All rights reserved. Versa Networks Confidential
TCPDUMP Logs from vni-0/302 & 303

admin@uCPE-Marriot-PaloAlto-cli> tcpdump vni-0/303 filter "host 172.16.125.90"

Starting capture on vni-0/303
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on _vni_0_303, link-type EN10MB (Ethernet), capture size 262144 bytes
13:10:22.887332 52:54:00:35:35:ae > 56:48:4f:53:54:01, ethertype IPv4 (0x0800), length 98: 172.16.125.90 > 8.8.8.8: ICMP echo request, id 5465, seq 6, length 64
13:10:22.899333 56:48:4f:53:54:01 > 52:54:00:35:35:ae, ethertype IPv4 (0x0800), length 98: 8.8.8.8 > 172.16.125.90: ICMP echo reply, id 5465, seq 6, length 64
13:10:23.895333 52:54:00:35:35:ae > 56:48:4f:53:54:01, ethertype IPv4 (0x0800), length 98: 172.16.125.90 > 8.8.8.8: ICMP echo request, id 5465, seq 7, length 64
13:10:23.911334 56:48:4f:53:54:01 > 52:54:00:35:35:ae, ethertype IPv4 (0x0800), length 98: 8.8.8.8 > 172.16.125.90: ICMP echo reply, id 5465, seq 7, length 64
^C
4 packets captured
admin@uCPE-Marriot-PaloAlto-cli> tcpdump vni-0/302 filter "host 172.16.125.90"
Starting capture on vni-0/302
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on _vni_0_302, link-type EN10MB (Ethernet), capture size 262144 bytes
13:10:28.935336 56:48:4f:53:54:00 > 52:54:00:a5:8b:c9, ethertype IPv4 (0x0800), length 98: 172.16.125.90 > 8.8.8.8: ICMP echo request, id 5465, seq 12, length 64
13:10:29.943333 56:48:4f:53:54:00 > 52:54:00:a5:8b:c9, ethertype IPv4 (0x0800), length 98: 172.16.125.90 > 8.8.8.8: ICMP echo request, id 5465, seq 13, length 64
13:10:30.951334 56:48:4f:53:54:00 > 52:54:00:a5:8b:c9, ethertype IPv4 (0x0800), length 98: 172.16.125.90 > 8.8.8.8: ICMP echo request, id 5465, seq 14, length 64
13:10:31.959335 56:48:4f:53:54:00 > 52:54:00:a5:8b:c9, ethertype IPv4 (0x0800), length 98: 172.16.125.90 > 8.8.8.8: ICMP echo request, id 5465, seq 15, length 64
^C
4 packets captured

Palo Alto Firewall not forwarding ICMP reply towards FlexVNF vni0-302 interface!

5 © 2017 Versa and/or its affiliates. All rights reserved. Versa Networks Confidential
Scenario 1b: Post implementing Policy Based Forwarding on Palo Alto, Remote SD-WAN branches started communicating with Internet

Solution implemented:

1. Configured PBF in Palo Alto for following Subnets:

a. Created a rule where any packet(mostly a response packet from Internet) hitting to Untrust/WAN interface and destined to SDWAN Branches will be forwarded to Trust interface ( in
order to bypass static routing)

6 © 2017 Versa and/or its affiliates. All rights reserved. Versa Networks Confidential
 MPLS 8.8.8.8
Server
INTERNET
172.16.125.90
Remote SDWAN Branch WAN IP: 192.168.2.1/24
Datapath a packet takes from Remote SDWAN Branch to Internet & Vice-Versa
WAN IP: 192.168.2.6/24
Vni0/0.0

Internet-Transport-VR

Return traffic
tvi0/602.0 W-ST-Marriott-LAN-VR-PK-INET
Internet Destined traffic

dtvi-0/41 Versa FlexVNF Firewall

tvi0/11.0
Palo Alto Firewall
DIA TRAFFIC PATH
Marriott-Control-VR/MP-BGP/Tunnels

mpls-vpn-core-instance

tvi0/603.0 L-ST-Marriott-LAN-VR-PK-INET

Palo Alto VM

172.16.20.2 Untrust
VRF: Marriott-LAN-VR Vni-0/303.0 172.16.20.1

Vni-0/302.0 172.16.10.1 172.16.10.2 Trust

parbhat@parbhat-Standard-PC-i440FX-PIIX-1996:~$ traceroute 8.8.8.8

Lan: 172.16.191.1/24 Vni0/1.0
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
1 172.16.125.1 (172.16.125.1) 6.785 ms 6.755 ms 6.746 ms
2 ***
3 172.16.10.2 (172.16.10.2) 100.968 ms 88.016 ms 87.945 ms –Palo Alto
4 192.168.2.6 (192.168.2.6) 87.865 ms 87.974 ms 87.945 ms – INET-VR
5 192.168.2.1 (192.168.2.1) 93.926 ms 100.814 ms 100.799 ms 172.16.191.5/24
6 10.0.0.1 (10.0.0.1) 100.788 ms 94.811 ms 94.863 ms
…
17 google-public-dns-a.google.com (8.8.8.8) 126.496 ms 107.481 ms 123.527 ms
7
parbhat@parbhat-Standard-PC-i440FX-PIIX-1996:~$ © 2017 Versa and/or its affiliates. All rights reserved. Versa Networks Confidential
TCPDUMP Logs from vni-0/302 & 303
admin@uCPE-Marriot-PaloAlto-cli> tcpdump vni-0/303 filter "host 172.16.125.90"
Starting capture on vni-0/303
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on _vni_0_303, link-type EN10MB (Ethernet), capture size 262144 bytes
14:43:18.095334 52:54:00:35:35:ae > 56:48:4f:53:54:01, ethertype IPv4 (0x0800), length 98: 172.16.125.90 > 8.8.8.8: ICMP echo request, id 4192, seq 1, length 64
14:43:18.115336 56:48:4f:53:54:01 > 52:54:00:35:35:ae, ethertype IPv4 (0x0800), length 98: 8.8.8.8 > 172.16.125.90: ICMP echo reply, id 4192, seq 1, length 64
14:43:19.099333 52:54:00:35:35:ae > 56:48:4f:53:54:01, ethertype IPv4 (0x0800), length 98: 172.16.125.90 > 8.8.8.8: ICMP echo request, id 4192, seq 2, length 64
14:43:19.127334 56:48:4f:53:54:01 > 52:54:00:35:35:ae, ethertype IPv4 (0x0800), length 98: 8.8.8.8 > 172.16.125.90: ICMP echo reply, id 4192, seq 2, length 64
^C
4 packets captured

admin@uCPE-Marriot-PaloAlto-cli> tcpdump vni-0/302 filter "host 172.16.125.90"

Starting capture on vni-0/302
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on _vni_0_302, link-type EN10MB (Ethernet), capture size 262144 bytes
14:44:43.143334 56:48:4f:53:54:00 > 52:54:00:a5:8b:c9, ethertype IPv4 (0x0800), length 98: 172.16.125.90 > 8.8.8.8: ICMP echo request, id 4194, seq 1, length 64
14:44:43.163333 52:54:00:a5:8b:c9 > 56:48:4f:53:54:00, ethertype IPv4 (0x0800), length 98: 8.8.8.8 > 172.16.125.90: ICMP echo reply, id 4194, seq 1, length 64
14:44:44.143333 56:48:4f:53:54:00 > 52:54:00:a5:8b:c9, ethertype IPv4 (0x0800), length 98: 172.16.125.90 > 8.8.8.8: ICMP echo request, id 4194, seq 2, length 64
14:44:44.167332 52:54:00:a5:8b:c9 > 56:48:4f:53:54:00, ethertype IPv4 (0x0800), length 98: 8.8.8.8 > 172.16.125.90: ICMP echo reply, id 4194, seq 2, length 64

8 © 2017 Versa and/or its affiliates. All rights reserved. Versa Networks Confidential
Debugs taken against each ICMP Request/Reply:
2019-02-14 15:07:15.883 DEBUG [0x103] vs_sfc_fltr_lookup_service_chain_id:1470 Ten 2 Sess hdl 0x0 Proto 1 SRC 172.16.125.90:4214 DST 8.8.8.8:4214 Fltr lkup rc 0 svc chain 128511
2019-02-14 15:07:15.883 DEBUG [0x103] vs_nfp_sess_create:2146 vs_nfp_sess_create, s-hdl 0 sess:0x7f42ed51aa00[0x00000000] flow[0]:(nil). flow[1]:(nil)

2019-02-14 15:07:15.883 DEBUG [0x103] vs_nfp_gen_shdl_from_sptr:428 Generating vs_sess_hdl_t 0x7f42ed51aa03

2019-02-14 15:07:15.883 DEBUG [0x103] vs_nfp_flows_create:819 Flow create, sess:0x7f42ed51aa00[0x02005e1e], flow[0]: 0x7f42ed4d2380 flow tuple[src 172.16.125.90 : 4214 dst 8.8.8.8 : 4214. proto 1 vrf
11]
2019-02-14 15:07:15.883 DEBUG [0x103] vs_nfp_sess_create:2257 s-hdl 0x7f42ed51aa03 sess:0x7f42ed51aa00 flow[0]:0x7f42ed4d2380. flow[1]:0x7f42ed4d2400
2019-02-14 15:07:15.883 DEBUG [0x103] vs_nfp_sess_create:2393 created s-hdl 0x7f42ed51aa03 at 74061004
2019-02-14 15:07:15.883 DEBUG [0x103] vs_nfp_sess_set_idle_tmo_internal:3150
idle timeout set[7f42ed51aa03] idle_tmo: 10000 hard_tmo: 0
2019-02-14 15:07:15.883 DEBUG [0x103] sfc_dp_trace_next_node_lookup:1274 Ing SC 128511 ingr if 1062 Sess hdl 0x7f42ed51aa03 action External/4 egr 1054 VRF 11 addr 172.16.10.2
2019-02-14 15:07:15.883 DEBUG [0x103] sfc_dp_trace_next_node_lookup:1274 Ing SC 128511 ingr if 1056 Sess hdl 0x7f42ed51aa03 action Local/0
2019-02-14 15:07:15.883 DEBUG [0x103] vs_nfp_add_nat_flow:1570 sess: 0x7f42ed51aa00[0x02005e1e] caller: 0xd3013d flow tuple[src 192.168.2.6 : 39956 dst 8.8.8.8 : 39956. proto 1 vrf 12]
2019-02-14 15:07:15.883 DEBUG [0x103] vs_nfp_nat_flow_create:261 NAT Flow Create sess:0x7f42ed51aa00[0x02005e1e], nat flow: 0x7f42ed4d2480
2019-02-14 15:07:15.902 DEBUG [0x103] sfc_dp_trace_next_node_lookup:1274 Ing SC -2147355137 ingr if 1046 Sess hdl 0x7f42ed51aa03 action Local/0
2019-02-14 15:07:15.902 DEBUG [0x103] sfc_dp_trace_next_node_lookup:1274 Egr SC -2147355137 ingr if 1046 Sess hdl 0x7f42ed51aa03 action External/4 egr 1056 VRF 11 addr 172.16.20.2

9 © 2017 Versa and/or its affiliates. All rights reserved. Versa Networks Confidential
 MPLS 8.8.8.8
Server
10.0.8.5 INTERNET
Remote SDWAN Branch
Comparison with previous datapath flow:
WAN IP: 192.168.2.1/24
Datapath a packet takes between Remote Branch/Internet & LAN-VR
Vni0/0.0
Return traffic
Internet-Transport-VR
SDWAN/Internet originated traffic

tvi0/602.0 W-ST-Marriott-LAN-VR-PK-INET
Versa FlexVNF Firewall

Palo Alto Firewall

dtvi-0/41

tvi0/11.0
DIA TRAFFIC PATH
Marriott-Control-VR/MP-BGP/Tunnels

mpls-vpn-core-instance

tvi0/603.0 L-ST-Marriott-LAN-VR-PK-INET
Palo Alto VM

172.16.20.2 Untrust
VRF: Marriott-LAN-VR Vni-0/303.0 172.16.20.1

Vni-0/302.0 172.16.10.1 172.16.10.2 Trust

Lan: 172.16.191.1/24 Vni0/1.0

admin@Marriot-Branch2-AWS-cli> traceroute 172.16.191.5 routing-instance Marriott-LAN-VR

traceroute to 172.16.191.5 (172.16.191.5), 30 hops max, 60 byte packets
1 ***
2 172.16.20.2 141.519 ms 141.501 ms 141.477 ms
3 172.16.191.1 141.437 ms 141.437 ms 141.437 ms
4 172.16.191.5 147.007 ms 189.745 ms 189.716 ms
[ok][2019-02-09 10:48:53] 172.16.191.5/24
admin@Marriot-Branch2-AWS-cli>
10 © 2017 Versa and/or its affiliates. All rights reserved. Versa Networks Confidential
TCPDUMP Logs from vni-0/302 & 303

admin@uCPE-Marriot-PaloAlto-cli> tcpdump vni-0/303 filter "host 172.16.125.90"

Starting capture on vni-0/303
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on _vni_0_303, link-type EN10MB (Ethernet), capture size 262144 bytes
11:55:12.407333 56:48:4f:53:54:01 > 52:54:00:35:35:ae, ethertype IPv4 (0x0800), length 98: 172.16.125.90 > 172.16.191.5: ICMP echo request, id 5437, seq 1, length 64
11:55:12.407345 52:54:00:35:35:ae > 56:48:4f:53:54:01, ethertype IPv4 (0x0800), length 98: 172.16.191.5 > 172.16.125.90: ICMP echo reply, id 5437, seq 1, length 64
11:55:13.403336 56:48:4f:53:54:01 > 52:54:00:35:35:ae, ethertype IPv4 (0x0800), length 98: 172.16.125.90 > 172.16.191.5: ICMP echo request, id 5437, seq 2, length 64
11:55:13.407333 52:54:00:35:35:ae > 56:48:4f:53:54:01, ethertype IPv4 (0x0800), length 98: 172.16.191.5 > 172.16.125.90: ICMP echo reply, id 5437, seq 2, length 64
11:55:14.407334 56:48:4f:53:54:01 > 52:54:00:35:35:ae, ethertype IPv4 (0x0800), length 98: 172.16.125.90 > 172.16.191.5: ICMP echo request, id 5437, seq 3, length 64
11:55:14.407338 52:54:00:35:35:ae > 56:48:4f:53:54:01, ethertype IPv4 (0x0800), length 98: 172.16.191.5 > 172.16.125.90: ICMP echo reply, id 5437, seq 3, length 64
^C
6 packets captured

admin@uCPE-Marriot-PaloAlto-cli> tcpdump vni-0/302 filter "host 172.16.125.90"

Starting capture on vni-0/302
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on _vni_0_302, link-type EN10MB (Ethernet), capture size 262144 bytes
11:54:22.503342 52:54:00:a5:8b:c9 > 56:48:4f:53:54:00, ethertype IPv4 (0x0800), length 98: 172.16.125.90 > 172.16.191.5: ICMP echo request, id 5435, seq 1, length 64
11:54:22.503356 56:48:4f:53:54:00 > 52:54:00:a5:8b:c9, ethertype IPv4 (0x0800), length 98: 172.16.191.5 > 172.16.125.90: ICMP echo reply, id 5435, seq 1, length 64
11:54:23.503349 52:54:00:a5:8b:c9 > 56:48:4f:53:54:00, ethertype IPv4 (0x0800), length 98: 172.16.125.90 > 172.16.191.5: ICMP echo request, id 5435, seq 2, length 64
11:54:23.503363 56:48:4f:53:54:00 > 52:54:00:a5:8b:c9, ethertype IPv4 (0x0800), length 98: 172.16.191.5 > 172.16.125.90: ICMP echo reply, id 5435, seq 2, length 64
11:54:24.503343 52:54:00:a5:8b:c9 > 56:48:4f:53:54:00, ethertype IPv4 (0x0800), length 98: 172.16.125.90 > 172.16.191.5: ICMP echo request, id 5435, seq 3, length 64
11:54:24.503356 56:48:4f:53:54:00 > 52:54:00:a5:8b:c9, ethertype IPv4 (0x0800), length 98: 172.16.191.5 > 172.16.125.90: ICMP echo reply, id 5435, seq 3, length 64
^C
6 packets captured

11 © 2017 Versa and/or its affiliates. All rights reserved. Versa Networks Confidential
Debugs taken against each ICMP Request/Reply:
2019-02-15 11:51:47.523 DEBUG [0x102] vs_nfp_gen_shdl_from_sptr:428 Generating vs_sess_hdl_t 0x7f42ed11b902
2019-02-15 11:51:47.523 DEBUG [0x102] vs_nfp_flows_create:819 Flow create, sess:0x7f42ed11b900[0x0200a7e5], flow[0]: 0x7f42ed0d2600 flow tuple[src 172.16.125.90 : 5432 dst 172.16.191.5 : 5432. proto 1
vrf 11]
2019-02-15 11:51:47.523 DEBUG [0x102] vs_nfp_sess_create:2257 s-hdl 0x7f42ed11b902 sess:0x7f42ed11b900 flow[0]:0x7f42ed0d2600. flow[1]:0x7f42ed0d2680
2019-02-15 11:51:47.523 DEBUG [0x102] vs_nfp_sess_create:2393 created s-hdl 0x7f42ed11b902 at 148751217
2019-02-15 11:51:47.523 DEBUG [0x102] vs_nfp_sess_set_idle_tmo_internal:3150
idle timeout set[7f42ed11b902] idle_tmo: 10000 hard_tmo: 0
2019-02-15 11:51:47.523 DEBUG [0x102] sfc_dp_trace_next_node_lookup:1274 Ing SC 128767 ingr if 1062 Sess hdl 0x7f42ed11b902 action External/4 egr 1056 VRF 11 addr 172.16.20.2
2019-02-15 11:51:47.523 DEBUG [0x102] sfc_dp_trace_next_node_lookup:1274 Ing SC 128767 ingr if 1054 Sess hdl 0x7f42ed11b902 action Local/0
2019-02-15 11:51:47.524 DEBUG [0x102] sfc_dp_trace_next_node_lookup:1274 Ing SC -2147354881 ingr if 1048 Sess hdl 0x7f42ed11b902 action Local/0
2019-02-15 11:51:47.524 DEBUG [0x102] sfc_dp_trace_next_node_lookup:1274 Egr SC -2147354881 ingr if 1048 Sess hdl 0x7f42ed11b902 action External/4 egr 1054 VRF 11 addr 172.16.10.2
2019-02-15 11:51:47.524 DEBUG [0x102] sfc_dp_trace_next_node_lookup:1274 Ing SC -2147354881 ingr if 1056 Sess hdl 0x7f42ed11b902 action Fwd/5
2019-02-15 11:51:48.524 DEBUG [0x102] sfc_dp_trace_next_node_lookup:1274 Ing SC 128767 ingr if 1062 Sess hdl 0x7f42ed11b902 action External/4 egr 1056 VRF 11 addr 172.16.20.2
2019-02-15 11:51:48.524 DEBUG [0x102] sfc_dp_trace_next_node_lookup:1274 Ing SC 128767 ingr if 1054 Sess hdl 0x7f42ed11b902 action Local/0
2019-02-15 11:51:48.525 DEBUG [0x102] sfc_dp_trace_next_node_lookup:1274 Ing SC -2147354881 ingr if 1048 Sess hdl 0x7f42ed11b902 action Local/0
2019-02-15 11:51:48.525 DEBUG [0x102] sfc_dp_trace_next_node_lookup:1274 Egr SC -2147354881 ingr if 1048 Sess hdl 0x7f42ed11b902 action External/4 egr 1054 VRF 11 addr 172.16.10.2
2019-02-15 11:51:48.525 DEBUG [0x102] sfc_dp_trace_next_node_lookup:1274 Ing SC -2147354881 ingr if 1056 Sess hdl 0x7f42ed11b902 action Fwd/5

12 © 2017 Versa and/or its affiliates. All rights reserved. Versa Networks Confidential
 Thank You

13 © 2017 Versa and/or its affiliates. All rights reserved. Versa Networks Confidential