Secure SD-WAN Deployment Guide For MSSPs

Secure SD-WAN
Deployment Guide for MSSPs

Version 6.4
FORTINET DOCUMENT LIBRARY
https://docs.fortinet.com
FORTINET VIDEO GUIDE

https://video.fortinet.com
FORTINET BLOG
https://blog.fortinet.com
CUSTOMER SERVICE & SUPPORT

https://support.fortinet.com
FORTINET TRAINING & CERTIFICATION PROGRAM

https://www.fortinet.com/training-certification
NSE INSTITUTE
https://training.fortinet.com
FORTIGUARD CENTER
https://www.fortiguard.com
END USER LICENSE AGREEMENT

https://www.fortinet.com/doc/legal/EULA.pdf
FEEDBACK
Email: techdoc@fortinet.com
November 4, 2021
Secure SD-WAN 6.4 Deployment Guide for MSSPs
01-646-749256-20211104
TABLE OF CONTENTS
Change Log 5
Introduction 6
Executive summary 6
Audience 6
About this guide 6
Design overview 8
Use cases and topologies 8
Single Hub topology 8
Dual Hub topology 9
Multi-Regional topology 10
Design concepts and considerations 11
Model devices 12
Templates 12
Automation 13
Multitenancy 13
Deployment overview 15
Product prerequisites 15
Deployment plan 15
FortiManager common elements 16
Configuring ADOMs 16
Configuring the root ADOM 16
Enabling and creating ADOMs 17
Creating device groups 19
Creating a default system template 19
Creating certificate templates 20
Creating normalized interfaces 21
Defining the corporate LAN object 22
Single Hub deployment 24
Configuring the Hub device 24
Configuring overlay (Hub) 24
Configuring routing (Hub) 29
Configuring SD-WAN (Hub) 32
Configuring firewall policies (Hub) 41
Deploying the Hub device 43
Configuring Edge devices 51
Configuring overlay (Edge) 51
Configuring routing (Edge) 55
Configuring SD-WAN (Edge) 58
Configuring firewall policies (Edge) 67
Deploying Edge devices 70
Verifying configuration 76
Secure SD-WAN 6.4 Deployment Guide for MSSPs 3

Fortinet Technologies Inc.
Dual Hub deployment 80
Configuring Hub devices 80
Configuring overlay 81
Configuring routing 85
Configuring SD-WAN template 87
Configuring firewall policies 92
Deploying Edge devices 92
Multi-Regional deployment 96
Addressing scheme 96
Deploying individual regions 96
Interconnecting regions 98
Configuring overlay and routing 98
Configuring firewall policies 109
Configuring Edge devices with SD-WAN 111
Additional topics 114
Signaling SLA status to hubs 114
Configuring Hub devices 118
SD-WAN routing logic 122
Using the default route via underlay 123
Disabling route check in SD-WAN rules 124
Excluding Traffic from SD-WAN 125
Appendix A - Products used 126

Change Log
Date Change Description
2021-11-04 Initial release.

Introduction
This document describes Fortinet's recommended approach to configuring our Secure SD-WAN Solution for Managed
Service Providers. It covers configuration of the basic building blocks of the solution and shows how they can be
extended to more complex deployments.
Before following this document, we recommend that you study our SD-WAN / SD-Branch Reference Architecture for
MSSPs.
This section contains the following topics:
l Executive summary on page 6
l Audience on page 6
l About this guide on page 6
Executive summary
Fortinet's solution consists of fully-functional FortiGate devices (FGT) deployed on every site and centrally managed by
FortiManager (FMG). Naturally, this offers a wide variety of deployment methods and configuration options. Using our
recommended configuration will ensure that your deployment follows the most widely tested best practices, resulting in a
highly scalable and easy-to-operate Secure SD-WAN Solution!
Audience
This guide is intended for a technical audience, including system architects and design engineers who want to deploy
Fortinet Secure SD-WAN or Secure SD-Branch in a managed offering capacity.
The solutions in this guide are targeted to Service Providers.
This guide assumes the reader is familiar with the basic concepts of applications, networking, routing, security, and high
availability, and has a basic understanding of network and data center architectures. For implementation, a working
knowledge of FortiOS networking and policy configuration is ideal.
About this guide
This deployment guide describes the design and deployment steps for a specific architecture. Readers should first
evaluate their environment to determine whether the architecture and design outlined in this guide is suitable for them. It
is advisable to review the SD-WAN / SD-Branch Architecture for MSSPs Reference Architecture Guide(s), if readers are
still in the process of selecting the right architecture. Readers can find these documents on the Fortinet Docs Library.
This document will outline the configuration steps and procedures to successfully deploy three of the most common SD-
WAN use cases. We will start by prepping FortiManager, which is the Fortinet central management solution that is

Introduction
designed around multitenancy. From here, we will build out each of the three SD-WAN designs mentioned below
sequentially. Each section requires full completion of the previous section in order to successfully configure.
You must complete each section before moving on to subsequent sections.

Design overview
The following sections help describe the design of the managed Fortinet Secure SD-WAN solution:
l Use cases and topologies on page 8
l Design concepts and considerations on page 11
Use cases and topologies
This document covers the following use cases and topologies for the managed Fortinet Secure SD-WAN Solution:
l Single Hub topology on page 8
l Dual Hub topology on page 9
l Multi-Regional topology on page 10
Single Hub topology
Single Hub SD-WAN is the fundamental building block of our solution. It is a simple Hub-and-Spoke topology with two
overlays, a Hub and two Spokes, with ADVPN enabled:

Design overview
The more advanced Multi-Hub and Multi-regional topologies are also covered in this guide, and are extensions of the
basic topology.
Overlay and routing configuration are generic and do not depend on a particular traffic steering
strategy that you would like to implement. They form a foundation of your solution, allowing all
types of traffic (Edge-to-Edge, Edge-to-Hub, Hub-to-Edge, Edge-to-Internet and Hub-to-
Internet). This is in accordance with the Five-Pillar Design Approach described in the SD-
WAN / SD-Branch Reference Architecture for MSSPs. It is the SD-WAN pillar that will define
the exact traffic steering strategy.
Dual Hub topology
Building on the single Hub SD-WAN topology, the second Hub is added in the same way the first was. In this guide, we
will extend the single Hub configuration to include two Hubs operating as Primary/Secondary:

Design overview
Multi-Regional topology
This section describes how to extend the topology to multiple regions:

Design overview
Although we will show an example with just two regions, you can use the same procedure to add additional regions to the
deployment.
Design concepts and considerations
The managed Fortinet Secure SD-WAN Solution described in this document consists of fully-functional FortiGate
devices (FGT) deployed on every site, and centrally managed by FortiManager (FMG) and FortiAnalyzer (FAZ).
Multiple deployment methods are supported. One option could be the following:
1. Manually deploy a new FGT with basic underlay configuration that is necessary to communicate with FMG.
2. Onboard the FGT to FMG by using the Device Discovery method, which is initiated from FMG.
3. Make the necessary configuration on FMG, and push it to the new FGT.
While this method can be satisfactory in some environments, we recommend following a different approach, which relies
on the concept of a Model Device in FMG. The Model Device behaves similar to any managed FGT device, except that
there is no real FGT device behind it. Therefore, the necessary steps will be as follows:
1. In FMG, create a Model Device for the new FGT that needs to be deployed.
2. In FMG, apply all the necessary configuration to the Model Device.
3. Link the real FGT device to the Model Device.
The first two steps are done locally on FMG, without any communication with the real FGT device. Hence, they can be
done during the preparation stage, when the real FGT device might not even exist yet.
The last step is where the configuration is pushed from FMG to the real device, making it part of the SD-WAN
deployment. This step can also be done in several ways:

Design overview
l Zero-Touch Provisioning (ZTP) using FortiDeploy lets you link the real FGT device to the Model Device with
minimal interaction with the FGT device itself: Once the new (unconfigured) FGT device is plugged in, it will call
home to obtain the location of the FMG
a. It will then contact FMG, which will authorize it, and map it to the corresponding Model Device by using itss
serial number.
b. FMG will then push the Model Device configuration to the FGT device.
c. Once this operation is complete, the FGT is fully deployed and managed by FMG.
l Zero-Touch Provisioning (ZTP) using DHCP Option is identical to the previous method, except for how the
location of FMG is obtained. Instead of calling home, the new (unconfigured) FGT device will receive this
information from the local DHCP server by using a special DHCP Option 240. But from the FMG perspective, the
sequence of events remains the same.
l Low-Touch Provisioning requires a customer to manually configure the FMG details on the new FGT device:
a. Once the new (unconfigured) FGT device is plugged in, the user connects to it, and manually enters details of
the FMG.
b. FGT then contacts FMG, and the rest remains identical to the previous methods.
Here again from the FMG perspective, the sequence of events remains the same.
Detailed instructions for the above methods are outside the scope of this document.
l Model devices on page 12
l Templates on page 12
l Automation on page 13
l Multitenancy on page 13
Model devices
By comparing to the traditional Device Discovery to the Model Device approach, you can see multiple advantages to
using the Model Device approach:
l The workflow does not depend on whether ZTP is used or not (or on what ZTP method is used). Different sites can
be onboarded by using different methods, but from the FMG perspective, the process remains the same for all of
them.
l Model Devices are local objects inside FMG. They can be safely deleted, if any configuration mistake has been
made. Since this configuration is not pushed to any real FGT device until the very last step, there is no need for
rollback.
l For the same reason, working with the Model Device approach is safer. Interim configuration will not cause service
disruption, since only the final configuration is pushed to the real FGT device at the very last step.
For the above reasons, we will be focusing on this approach throughout this document.
Templates
A Model Device behaves similarly to any managed device in FMG. Therefore, the configuration could be done on a per-
device basis by using Device Manager.

Design overview
We strongly discourage you from using this method when configuring your Secure SD-WAN
Solution! Interactive per-device configuration using Device Manager quickly becomes
unacceptable, as the number of sites grow. It cannot be easily replicated to other devices (or to
other environments).
For this reason, we recommend relying on Templates assigned to multiple devices. These include Provisioning
Templates, SD-WAN Templates, and CLI Templates. This document describes all of these templates in more detail.
With templates, the amount of ad-hoc per-device configuration should be reduced to the minimum: it should be limited
mainly to setting per-device Meta Fields and assigning the right set of Templates.
When a template cannot be used to specify certain parts of the configuration, you should apply the configuration by using
CLI Scripts. CLI Scripts are preferred over interactive per-device configuration, since they provide better tracking.
We would like to highlight that using Templates is always encouraged, whether you are
planning to use Zero-Touch Provisioning or not. Good reusable Templates are the key for a
successful large-scale Secure SD-WAN deployment!
Automation
FortiManager (FMG) provides a comprehensive REST API that allows complete automation of the deployment. Every
action described in this document can be automated with an API call made using an automation framework of your
choice, such as Ansible, Python, and so on.
As a result, it is possible to deploy our entire Secure SD-WAN Solution in a fully automated, unattended manner. With a
fully automated deployment, there is no need to log in to FMG.
While a detailed walkthrough is outside the scope of this document, it is useful to highlight some of the benefits of an
automated deployment:
l Infrastructure as Code (IaC). All templates and other FMG objects can be imported from external repository by
using automation. This lets you quickly replicate the environment from scratch, for example, to maintain consistency
between staging and production environments.
l Model Device creation can be automated as well, including filling in the right Meta Field values per site, assigning
the right set of Templates, and performing any other required configuration. For example, Model Device creation
can be triggered by an external Onboarding Portal, to which the remote site operator connects, in order to fill in the
necessary details of the new FGT device (such as its Serial Number).
l For Managed Secure Service Providers (MSSPs), custom Onboarding Portals provide a valuable opportunity to
speak their end-customer’s language: the UI of those Portals can be designed with consideration to a particular
end-customer’s business, using the relevant language, rather than generic SD-WAN terminology.
l Needless to say, automation saves time, minimizes human mistakes, and becomes indispensable in large-scale
deployments.
Multitenancy
Each component of our solution offers flexible multitenancy options. FortiGate devices (FGT) can be shared between
multiple tenants using Virtual Domains (VDOMs). FortiManager (FMG) and FortiAnalyzer (FAZ) use Administrative
Domains (ADOMs).

Design overview
When our solution is offered as a Managed Service, the following deployment options are typically considered by the
Service Providers (MSSPs):
l Edge FGTs are deployed on end-customer premises, and they are dedicated to those end-customers (no
multitenancy).
l Hub FGTs:
l Option 1: Deployed on MSSP premises, multitenant, VDOM per end-customer
l Option 2: Deployed on MSSP premises, dedicated to end-customer, no multitenancy (usually virtual)
l Option 3: Deployed on end-customer premises (Enterprise design), no multitenancy
l FMG/FAZ:
l Option 1: Deployed on MSSP premises, multitenant, ADOM per end-customer
l Option 2: Deployed on MSSP premises or in public cloud, dedicated to end-customer, no multitenancy (virtual)
When MSSP offers a fully managed service, end-customers might not even require access to the devices. But quite
often a certain level of access is provided. The following options are worth considering:
l If end-customers must have a certain level of access to FGT devices (often read-only), a dedicated customer
VDOM can be created. The management (“root”) VDOM will be accessible by the MSSP only.
l Central Management and Monitoring:
l End-customer may be granted direct FMG/FAZ access, to their respective ADOM only (or to the entire
instance, when it is dedicated to end-customer).

l Alternative option is to add FortiPortal to the solution. FortiPortal is a comprehensive self-service portal
designed for the end-customers.

l Yet another option is to use a custom MSSP Portal developed in-house. This portal can communicate to
FMG/FAZ by using REST API, thanks to the comprehensive automation support.

The choice of multitenancy model has minimal impact on the rest of this document. It is only important to make sure that
the configuration is done in the right ADOM.

Deployment overview
Deployment overview
The deployment overview includes the following sections:

l Product prerequisites on page 15
l Deployment plan on page 15
Product prerequisites
FortiGate prerequisites:
l Managed by FortiManager
FortiManager prerequisites:
l Managing FortiGate
l SD-WAN central management is enabled
For firmware versions, see Appendix A - Products used on page 126.
Deployment plan
Following is a high-level of how to configure the following products before deploying SD-WAN:
l Prepare FortiGate:
l Enable central management by FortiManager.
l Prepare FortiManager:
l Authorize management of FortiGate.
l Enable central SD-WAN management.

FortiManager common elements
This section sets up aspects of FortiManager to enable SD-WAN configuration in a scalable and consistent manner.
These common elements are reused and only need to be configured once.
Following is a summary of how to configure common elements:
1. Ensure ADOMs are configured to use FortiGate 6.4 and have SD-WAN central management enabled. See
Configuring ADOMs on page 16.
You can use a root ADOM for all end-customers, or you can enable ADOMs and create an ADOM for each end-
customer.
2. Create device groups for edge and hub devices. See Creating device groups on page 19.
Groups help to organize your view of devices in FortiManager, and enable you to assign templates to all devices in a
given group.
3. Create a System template. See Creating a default system template on page 19.
A system template controls some of the settings of the FortiGate itself, but does not define how it processes traffic.
4. Create a Certificate template. See Creating certificate templates on page 20.
A Certificate template defines how certificates are issued to the FortiGates. Certificates will be used in IPsec
authentication later in the guide.
5. Create Normalized interfaces. See Creating normalized interfaces on page 21.
Normalizing an interface associates an object interface, which exists only in FortiManager, with a physical interface
on the FortiGate.
6. Define a corporate LAN object. See Defining the corporate LAN object on page 22.
Define the subnet object for ease of reference in future policies, routing, and IPSec configuration.
Configuring ADOMs
You can use FortiManager with Administrative Domains (ADOMs) disabled or enabled. When ADOMs are disabled on
FortiManager, you can use one root ADOM for all end-customers. When ADOMs are enabled, you can define a separate
ADOM per end-customer.
Regardless of whether ADOMs are disabled or enabled, the deployment use cases described in this document use
ADOM version 6.4 with SD-WAN central management enabled.
This topic contains the following sections:
l Configuring the root ADOM on page 16
l Enabling and creating ADOMs on page 17
Configuring the root ADOM
With ADOMs disabled, FortiManager is dedicated to a single end-customer, and you will use the default, pre-existing
root ADOM.

To configure the root ADOM:
1. In FortiManager, go to System Settings > All ADOMs.

2. Select the root ADOM, and click Edit.
3. Beside Central Management, select SD-WAN, and click OK.
Enabling and creating ADOMs
When you enable ADOMs, you can define a separate ADOM per end-customer in FortiManager.
Ensure that each ADOM is created for FortiGate version 6.4, and SD-WAN central management is enabled.
To create an ADOM:
1. In FortiManager, enable ADOMs:

a. Log in to FortiManager as a super user administrator.
b. Go to System Settings > All ADOMs.
c. In the toolbar, select Enable ADOM.
You are automatically logged out of FortiManager, and returned to the log in screen.
2. Log in to FortiManager.
The Select an ADOM window is displayed.

3. Click the Create New button.

The Create New ADOM pane is displayed.
4. In the Name box, type a name for the ADOM.

5. In the Type box, select FortiGate, and click 6.4.

6. Beside Central Management, select SD-WAN.
7. Set the remaining options as desired, and click OK.
The ADOM is created.
Creating device groups
In Device Manager, create the following device groups:

l Edge
l Hubs
To create device groups:
1. In FortiManager, go to Device Manager > Device & Groups.

2. From the Device Group menu, select Create New Group.
3. In the Group Name box, type Hubs, and click OK. The device group is created.
4. Repeat this procedure to create a device group named Edge.
Creating a default system template
Create a system template with the basic settings for your environment. In our example, we use this template to configure
DNS servers and FortiAnalyzer logging.
To create a system template:
1. In FortiManager, go to Device Manager > Provisioning Templates > System Templates.

2. From the Create New menu, select Default Template.
The Create Default System Template dialog box is displayed.
3. In the Name box, type a name for the template, and click OK.
The template is created.

4. Double-click the template to open it for editing.

5. Delete all widgets, except for the widgets that you want to use, such as the DNS widget and the Log Settings widget.
a. Click the top-right corner of each unneeded widget, and select Delete.
b. In the confirmation dialog box, click OK.
6. In the DNS widget, enter settings for your environment, and click Apply.
7. (Optional) If sending logs to FortiAnalyzer, in the Log Settings widget, enter settings for your environment, and click
Apply.
Creating certificate templates
This section describes how to create the following certificate templates:

l Edge certificate template
l Hub certificate template
Later we will use these certificate templates to issue local certificates for all devices, and the certificates will be used for
IPSEC authentication.
To create a certificate template:
1. In FortiManager, go to Device Manager > Provisioning Templates > Certificate Templates, and click Create New.
The Create New Certificate Template pane is displayed.
2. Beside Type, select Local.
This lets you use FMG built-in Certificate Authority (CA) for issuing certificates.

3. In the Certificate Name box, type Edge.
4. Complete the remaining options as desired for your environment, and click OK.
A certificate template named Edge is created.
5. Repeat this procedure to create a certificate template named Hubs.
Creating normalized interfaces
Create the following normalized interfaces:
Name Mapping Notes
lan port5 LAN-facing interface on all devices
lo-HC lo-HC Health-check loopback on Hubs
Map the LAN-facing interface according to your environment. We recommend using Per-Platform Mapping.
Normalizing an interface associates an object interface, which exists only in FortiManager, with a physical interface on
the FortiGate.
To create normalized interfaces:
1. In FortiManager, go to Policy & Objects > Object Configurations > Normalized Interface.
2. Click Create New.
The Create New Normalized Interface pane is displayed.
3. In the Name box, type a name, such as lan.

4. Add a per-platform mapping for lan.

a. Under Per-Platform Mapping, click Create New.
The Create new Per-Platform Mapping dialog box is displayed.
b. In the Matched Platform list, select all.
When you select all, a default mapping rule is created.
c. In the Mapped Interface Name box, type port5.

d. Click OK.
The per-platform mapping is created.
5. Click OK.
The normalized interface is created.
6. Repeat this procedure to create a normalized interface named lo-HC.
Defining the corporate LAN object
Create an Address object for the corporate network summary (such as an RFC1918 network).
You can reference the subnet object in future policies, routing, and IPSec configuration for ease of use.
To define the corporate LAN object:
1. In FortiManager, go to Policy & Objects > Object Configurations > Firewall Objects.
2. In the tree menu, select Addresses, and from the Create New menu, select Address.
The Create New Address pane is displayed.
3. In the Address Name box, type CORP_LAN.

4. Set the remaining options for your network, and click OK.

Single Hub deployment
In this chapter we will guide you through the configuration of the most fundamental building block of our solution. It is a
simple Hub-and-Spoke topology with two overlays, a Hub and two Spokes, with ADVPN enabled.
Before you begin, ensure that you have completed the following tasks:
1. Complete common elements. See FortiManager common elements on page 16.
Following is a summary of how to configure Single Hub SD-WAN/ADVPN:
1. Configure and deploy the Hub device. See Configuring the Hub device on page 24.
2. Configure and deploy Edge devices. See Configuring Edge devices on page 51.
3. Verify the configuration. See Verifying configuration on page 76.
See also Single Hub topology on page 8.
Configuring the Hub device
Following is a summary of how to configure the hub device for Single Hub SD-WAN/ADVPN:
1. Configure overlays. See Configuring overlay (Hub) on page 24.
2. Configure routing. See Configuring routing (Hub) on page 29.
3. Configure SD-WAN. See Configuring SD-WAN (Hub) on page 32.
4. Configure the firewall policy. See Configuring firewall policies (Hub) on page 41.
5. Deploy the hub device. See Deploying the Hub device on page 43.
Configuring overlay (Hub)
The Overlay defines the different options traffic has to reach its destination. In this case, we will use IPSEC tunnels to get
traffic to the Hub.
Following is a summary of how to configure overlays:
1. Create a CLI Template. See Creating a CLI template on page 24.
This CLI template will configure Dial-Up IPSEC endpoints on the Hub, to which all the Edge devices will establish
static IPSEC tunnels.
2. Create CLI Template Group and add the above CLI Template. See Creating a CLI template group on page 27.
Template groups allow for several CLI Templates to be assigned to a FortiGate.
3. Define Meta Fields. See Defining Meta Fields on page 28.
Defining Meta Fields creates new fields on FortiGates. These are populated when creating the model device.
Creating a CLI template
Create a CLI template called 01-Hub-Overlay.

This CLI template will configure Dial-Up IPSEC endpoints on the Hub, to which all the Edge devices will establish static
IPSEC tunnels. You should follow the following guidelines:
l Create a separate Dial-Up IPSEC endpoint for each desired overlay
l Use IKEv2 with Mode Config, in order to automatically assign tunnel IP addresses to the Edge devices
l Enable certificate-based authentication (use “Hub” certificate, matching the name of the Certificate Template
created earlier)
l Disable add-route feature, since the routing will be handled by BGP
l Use the default net-device disable setting with tunnel-search nexthop (this combination is required for
correct operation of dynamic routing in conjunction with Dial-Up IPSEC endpoints)
l Enable ADVPN sender
l Enable network-overlay feature and define unique Network ID for each overlay
l Manually configure tunnel IP addresses for the Hub itself
l Use Meta Fields (variables), in order to make your CLI Template generic and applicable to multiple Hubs
Here is the content of this CLI Template in our example topology. It follows all the above guidelines:
config vpn ipsec phase1-interface
# Local Region (Edges)
edit "EDGE_INET"
set type dynamic
set interface $(inet-intf)
set ike-version 2
set authmethod signature
set peertype any
set mode-cfg enable
set proposal aes256-sha256
set net-device disable
set tunnel-search nexthop
set add-route disable
set auto-discovery-sender enable
set network-overlay enable
set network-id $(inet-id)
set ipv4-start-ip $(inet-tunnel-net:4,2)
set ipv4-end-ip $(inet-tunnel-net:4,253)
set ipv4-netmask $(tunnel-mask)
set certificate "Hub"
next
edit "EDGE_MPLS"
set type dynamic
set interface $(mpls-intf)
set ike-version 2
set peertype any
set mode-cfg enable
set net-device disable
set tunnel-search nexthop
set auto-discovery-sender enable
set network-id $(mpls-id)
set ipv4-start-ip $(mpls-tunnel-net:4,2)
set ipv4-end-ip $(mpls-tunnel-net:4,253)
set ipv4-netmask $(tunnel-mask)

next
end
edit "EDGE_INET"
set phase1name "EDGE_INET"
set keepalive enable
next
edit "EDGE_MPLS"
set phase1name "EDGE_MPLS"
next
end
config system interface
edit "EDGE_INET"
set vdom "root"
set ip $(inet-tunnel-net:4,1) 255.255.255.255
set allowaccess ping
set type tunnel
set remote-ip $(inet-tunnel-net:4,254) $(tunnel-mask)
next
edit "EDGE_MPLS"
set vdom "root"
set ip $(mpls-tunnel-net:4,1) 255.255.255.255
set type tunnel
set remote-ip $(mpls-tunnel-net:4,254) $(tunnel-mask)
next
edit "lo-HC"
set vdom "root"
set ip 10.200.99.1 255.255.255.255
set type loopback
next
end
config system settings
set tcp-session-without-syn enable
end
The following Meta Fields are used:
Meta Field Description
inet-intf Internet interface
inet-id Unique Network ID for the Internet overlay
inet-tunnel-net Tunnel subnet for the Internet overlay
mpls-intf MPLS interface
mpls-id Unique Network ID for the MPLS overlay
mpls-tunnel-net Tunnel subnet for the MPLS overlay
tunnel-mask Subnet mask for the overlays

In addition to the IPSEC configuration, the template defines a special loopback interface - lo-HC. It will be configured
with the same IP address on the Hub device (and multiple Hub devices if used in the future), and it will be used by the
Edge devices to monitor health of the overlays.
Finally, the template also enables tcp-session-without-syn feature. We will discuss it in more detail when we
configure firewall policies.
To create a CLI template:
1. In FortiManager, go to Device Manager > Provisioning Templates > CLI Template.

2. From the Create New menu, select CLI Template, and create a new template called 01-Hub-Overlay.
You can also import a template from an external file by using the Import CLI Template button.
Creating a CLI template group
Create a CLI template group called Hub-Template, and add the overlay CLI template to the CLI Template Group.
To create a CLI template group:

2. From the Create New menu, select CLI Template Group, and create a new CLI Template Group called Hub-
Template.

3. Add the 01-Hub-Overlay CLI template to the group.
Defining Meta Fields
Create the Meta Fields used in CLI templates.
To create Meta Fields:
1. In FortiManager, go to System Settings > Advanced > Meta Fields.

2. Click Create New in the toolbar.
The Create New Meta Fields pane is displayed.

3. Create the following Meta Fields:

For all Meta Fields, set Object to Device, and set Importance to Optional.
inet-id Unique Network ID for the Internet overlay
inet-tunnel-net Tunnel subnet for the Internet overlay
mpls-id Unique Network ID for the MPLS overlay
mpls-tunnel-net Tunnel subnet for the MPLS overlay
tunnel-mask Subnet mask for the overlays
Configuring routing (Hub)
Routing will be handled by a dynamic routing protocol, as necessary for ADVPN. BGP is chosen for this example.
Following is a summary of how to configure routing:
1. Create a CLI template for routing. See Creating a CLI template for routing on page 29.
This CLI template will configure dynamic BGP neighbor on the Hub, and all edge devices will establish iBGP
sessions with the Hub.
2. Add the CLI template to the existing CLI template group. See Adding the CLI template to the existing template group
on page 31.
Ensure the CLI template is listed after 01-Hub-Overlay because order is important.
3. Define Meta Fields for use in CLI templates. See Defining Meta Fields on page 32.
Creating a CLI template for routing
In this section, you'll create a CLI template called 02-Hub-Routing.

The 02-Hub-Routing CLI template will configure dynamic BGP neighbor (neighbor-group feature) to which all the Edge
devices will establish IBGP sessions. You should follow the following guidelines:
l Enable IBGP Multipath and ADD-PATH features.
l Enable IBGP Route Reflector.
l Advertise a LAN subnet(s) behind the Hub.
It is also required to configure an overlay stickiness policy, which are policy routes that instruct the Hub to preserve the
overlay end-to-end for Edge-to-Edge traffic, whenever possible.
These policy routes are crucial for a correct operation of ADVPN. During ADVPN shortcut
establishment, the Hub must forward shortcut queries between the two negotiating Edges, and
it relies on routing to do so. The shortcut that will be established eventually depends on the
routing decisions made by the Hub! Hence, it is important that it preserves the overlay
whenever possible. Otherwise there could be an attempt to build a shortcut between two
separated underlay transports (such as Internet and MPLS), which is physically impossible.

Here is the content of this CLI Template in our example topology, that follows all the above guidelines:
config router bgp
set as $(as)
set router-id $(lan-net:4,1)
set keepalive-timer 5
set holdtime-timer 15
set ibgp-multipath enable
set ebgp-multipath enable
set additional-path enable
# additional-path-select = max. number of overlays * max. number of hubs
set additional-path-select 2
config neighbor-group
edit "EDGE"
set soft-reconfiguration enable
set advertisement-interval 1
set link-down-failover enable
set next-hop-self enable
set remote-as $(as)
set additional-path send
# adv-additional-path = max. number of overlays * max. number of hubs
set adv-additional-path 2
set route-reflector-client enable
next
end
config neighbor-range
edit 1
set prefix 10.200.0.0/14
set neighbor-group "EDGE"
next
end
config network
edit 1
set prefix $(lan-net)
next
end
end
config router policy

edit 1
set input-device "EDGE_INET"
set output-device "EDGE_INET"
set dst 10.0.0.0/8
next
edit 2
set input-device "EDGE_MPLS"
set output-device "EDGE_MPLS"
set dst 10.0.0.0/8
next
end
config router static

edit 101
set dst 10.0.0.0/8
set blackhole enable
set comment "Avoid potential leak of corporate traffic to underlay"
next
edit 102

set dst 10.200.0.0/14

set comment "Avoid potential recursive resolution of corporate BGP routes via
underlay"
next
end
Note that the following summaries are hard-coded into the template for simplicity. You may need to review them and
adjust to your environment. Alternatively, they could be defined as additional Meta Fields:
Value Description
10.0.0.0/8 Corporate summary (all LAN prefixes in the corporate network)
10.200.0.0/14 Tunnel subnet summary (all overlay tunnels)
Value Description
as BGP Autonomous System
lan-net LAN subnet behind the Hub
lan-summary Summary of all LAN subnets (within the region)
The Meta Field lan-summary is currently not used in the CLI templates for the Hub. The Hub
will require it only when we extend the topology to multiple regions. On the Edge devices, on
the other hand, it is required even for a single-region deployment, as will be described later.
For the sake of a good order, we will always set its value.
To create a CLI template for routing:

2. From the Create New menu, select CLI Template, and create a new template called 02-Hub-Routing.
Adding the CLI template to the existing template group
Add the CLI template called 02-Hub-Routing to the CLI Template Group called Hub-Template.
To add the CLI template to the existing CLI template group:

2. Under CLI Template Group, double-click the Hub-Template to open it for editing.

3. Beside Members, click + to select the 02-Hub-Routing CLI Template, and add it to the group.
Define Meta Fields for use in CLI templates.
To define Meta Fields:

Value Description
lan-net LAN subnet behind the Hub
Configuring SD-WAN (Hub)
SD-WAN central management must be enabled in the ADOM to use SD-WAN templates. See
Configuring ADOMs on page 16.
Following is a summary of how to configure the SD-WAN template for Hub devices:
1. Define interface members. See Defining interface members on page 33.
2. Configure SD-WAN template. See Creating an SD-WAN template (Hub) on page 36.

3. Configure SD-WAN as the default route. See Configuring SD-WAN as default route (Hub) on page 40.
Defining interface members
Defining SD-WAN interfaces allows you to add them to zones, which can then be used in policies. SD-WAN interfaces
are also used when creating SD-WAN rules.
Following is a summary of how to define interface members:
1. Define SD-WAN interfaces for Edge devices for each of their overlay interfaces. See Defining interfaces for Edge
devices on page 33.
Edge devices will have two overlay interfaces: INET and MPLS.
2. Define an SD-WAN interface for the Hub device for the underlay interface. See Defining interfaces for the Hub
device on page 34.
The hub has one underlay interface, which is its internet-facing interface.
Defining interfaces for Edge devices
For the overlays, create the following Interface Members, and map them to the respective Dial-Up IPSEC endpoints on
the Hub device:
l EDGE_INET
l EDGE_MPLS
The following table summarizes the Interface Members you need to create for Edge devices:
Link Interface Member Name Mapping
Overlay EDGE_INET Map to the respective Dial-Up IPSEC

endpoints on the Hub device.
Overlay EDGE_MPLS
To define interface members:
1. Go to Device Manager > SD-WAN > Interface Members, and click Create New.
The SD-WAN tab is used for centralized SD-WAN management.
The Create New WAN Interface pane is displayed.

2. In the Name box, type EDGE_INET.

3. Under Advanced Options, set priority value to 10.

We give higher priority value (meaning lower priority) to all the overlay interfaces, so that Internet traffic uses
underlay interfaces by default. This includes all the locally originated Internet traffic (such as FortiGate
communication with FortiGuard services), as well as all the Internet traffic that is not explicitly controlled by SD-WAN
rules.
4. Create a new Normalized Interface named EDGE_INET:
a. In the Normalized Interface drop-down list, click the + (plus) icon to create a new Normalized Interface.
The Create New Normalized Interface dialog box is displayed.
b. In the Name box, type EDGE_INET
c. Under Per-Platform Mapping, click Create New.
d. In the Matched Platform list, select all.
e. In the Mapped Interface Name, type EDGE_INET.
This setting maps the Normalized Interface to the phase1-interface of the same name. The resulting SD-WAN
Interface Member will look as follows:
5. Repeat the same steps to create an Interface Member named EDGE_MPLS.
Defining interfaces for the Hub device
For the underlay, create an Interface Member named UL_INET, and map it to the respective Internet-facing interface on
the Hub.
The following table summarizes the Interface Members you need to create for the Hub device:

Link Interface Member Name Mapping
Underlay UL_INET Map it to the respective Internet-facing

interface on the Hub device.
To define interfaces for the Hub device:
1. Go to Device Manager > SD-WAN > Interface Members.

2. In the tree menu, select Interface Members, and click Create New.

3. In the Name box, type UL_INET.
4. Create a new Normalized Interface named UL_INET:
a. In the Normalized Interface drop-down list, click the + (plus) icon to create a new Normalized Interface.
The Create New Normalized Interface dialog box is displayed.
b. In the Name box, type UL_INET
c. Under Per-Platform Mapping, click Create New.
d. In the Matched Platform list, select all.
e. In the Mapped Interface Name, type UL_INET.
This setting maps the Normalized Interface to the Internet-facing interface on the Hub device. The resulting
SD-WAN Interface Member will look as follows:

Creating an SD-WAN template (Hub)
An SD-WAN template is used to deploy the SD-WAN configuration. As part of creating an SD-WAN template, you will
define SD-WAN zones, and add members to the zones.
This SD-WAN configuration does not cover Hub-to-Edge traffic (sessions originated behind
the Hub). As a result, if such traffic exists, it will be handled by the traditional routing,
disregarding the health of the overlays towards the Edge devices. If it is desired to let SD-WAN
select the optimal path for Hub-to-Edge traffic, please follow the steps detailed in Signaling
SLA status to hubs on page 114.

To configure an SD-WAN template:
1. In FortiManager, go to Device Manager > SD-WAN > SD-WAN Templates, and click Create New.
2. In the Name box, type Hub-Template.

3. Create SD-WAN zones named overlay and underlay:
a. Under Interface Members, click Create New > SD-WAN Zone.

b. In the Name box, type overlay, and click OK.
The SD-WAN zone named overlay is created.

c. Repeat this procedure to create an SD-WAN zone named underlay.
Keep the zones empty at the moment.
4. Create SD-WAN Members for this template, assigning them to the appropriate SD-WAN Zones, as follows:
Interface Member SD-WAN Zone
UL_INET underlay
EDGE_INET overlay
EDGE_MPLS overlay

a. Under Interface Members, click Create New > SD-WAN Member.
The Create New SD-WAN Interface Member dialog box is displayed.

b. In the Interface Member list, select UL_INET, and in the SD-WAN Zone list select underlay, and click OK.
c. Create the remaining interface members.

d. Click OK.
The interface members are created.

5. Click OK to save the SD-WAN Template.

No further configuration is required in the SD-WAN Template on the Hub at the moment.
Configuring SD-WAN as default route (Hub)
Configure the default route on the FortiGate to be your configured SD-WAN interface members using a CLI template.
Following is a summary of how to configure SD-WAN as a default route:
1. Create a CLI Template to configure the SD-WAN default route. See Creating a CLI template for the default route on
page 40.
2. Add the template to the existing group. See Adding the CLI template to the existing template group on page 41.
For a generic configuration, it is recommended that SD-WAN acts as a default route. This will automatically generate
FortiOS default route via all the configured SD-WAN Interface Members.
Configuring SD-WAN to act as a default route is not mandatory, but it ensures that SD-WAN
rules are fully responsible for traffic steering, making the routing configuration generic. For
more details, see SD-WAN routing logic on page 122.
Creating a CLI template for the default route
To configure SD-WAN as default route:

2. From the Create New menu, select CLI Template. The Create New CLI Template pane is displayed.
3. In the Name box, type 03-Default.
4. In the Script details box, paste the following content, and click OK.
edit 100
set sdwan enable
next
end

To add the template to the existing template group:

2. Under CLI Template Group, double-click the Hub-Template to open it for editing.
3. Beside Members, click + to select 03-Default and add it to the CLI template group named Hub-Template that we
created earlier:
Configuring firewall policies (Hub)
Policies control what traffic is permitted, and leverages the previously created Hub group and SD-WAN zones.
Following is a summary of how to configure firewall policies:
1. Create a new policy package named Hubs, and assign it to the group named Hub. See Creating a policy package
on page 41.
This automatically applies the policy package to any FortiGate in the Hub group.
2. Define five firewall policies in the Hubs policy package to permit traffic. See Defining policies on page 42.
These firewall policies leverage the SD-WAN zones and interfaces.
Creating a policy package
To create a policy package:
1. In FortiManager, go to Policy & Objects > Policy Packages.

2. From the Policy Package menu, select New.
The Create New Policy Package dialog box is displayed.
3. In the Name box, type Hubs, and click OK.
The policy package is created.

4. Under the policy package, select Installation Targets, and click Edit.
The Edit Installation Targets dialog box is displayed.
5. Select the Device Group named Hubs, and click OK to add its installation targets to the policy package:
Defining policies
To define policies:
1. In FortiManager, go to Policy & Objects > Policy Packages > Hubs > Firewall Policy.
2. Click Create New, and create the following Firewall Policies:
Name From To Src Dst Service NAT Action
Edge-Edge overlay overlay CORP_ CORP_ ALL No Accept

LAN LAN (see *)
Edge-Hub lan overlay lan overlay CORP_ CORP_ All No Accept

LAN LAN
Health- overlay lo-HC all all PING No Accept

check
Internet lan underlay all all ALL Yes Accept

(DIA)
Internet overlay underlay all all ALL Yes Accept

(RIA)
* For Edge-Edge rule, we must configure the following Advanced Options:
Parameter Value
anti-replay off
tcp-session-without-syn all
This is necessary to support existing TCP session switchover due to changes in SD-WAN steering decision:
l If the traffic flows via direct Edge-to-Edge tunnel (ADVPN shortcut), the session on the Hub remains idle, and
thus it will eventually timeout.

l Then, if Edge SD-WAN makes a decision to switchover to a different overlay (due to the change in network
conditions), the next few packets may need to flow via the Hub again.
l Since this TCP session no longer exists on the Hub, the traffic will be dropped.
l To avoid this, we configure the above options.
These options do not compromise the security, because they only apply to Edge-to-Edge traffic, which will be
protected by the Edge devices (the corresponding firewall rules will be covered later). The Edge devices will keep
performing complete stateful inspection of this traffic, whether it flows via the Hub or via a direct Edge-to-Edge
tunnel.
Notes:
l The SD-WAN Zones underlay and overlay were automatically created to be used in the Firewall Policy.
l This Firewall Policy is ready to support Remote Internet Access, which is traffic arriving from the Edge devices via
the overlays, destined to the Internet (underlay).
l This Firewall Policy also allows Direct Internet Access for the workloads hosted behind the Hub itself.
l We must explicitly allow health-check probes that the Edge devices will send to the Hub device.
l Any desired security inspection can be applied, although keep in mind that Edge-to-Edge traffic will be typically
secured by the Edge devices themselves.
Deploying the Hub device
This is the final step for the Hub deployment. All previous configuration is assigned and installed to the model device.
Once the model device is ready, the real device may register with FortiManager to receive its full configuration.
Following is a summary of how to deploy the Hub device:
1. Add a Model Device. See Adding a model device on page 44.
A model device is used as a placeholder, until the real device connects to the FortiManager.
2. Configure the Meta Fields. See Configuring Meta Fields on page 46.
Meta Field values are a one-time definition for each device as it is on boarded.
3. Generate the certificate for the Hub device. See Generating and issuing certificates for Hub devices on page 47.
This step leverages the previously defined Hub Certificate Template.
4. Install the overlay configuration to the Model Device. See Installing the overlay configuration on page 48.
When the real device connects, the model device configuration will be installed to the real device.
5. Install the remaining configuration to the Model Device. See Installing the remaining configuration to the model
device on page 50.
6. Connect the real device. See Connecting the real device on page 50.
You can connect the real device manually or by using zero-touch provisioning (ZTP).

Adding a model device
A model device is used as a placeholder, until the real device connects to the FortiManager.
To add a model device:
1. On Device Manager, click Add Device, and select Add Model Device.
2. In the Name box, type the name of the Hub, and select the right parameters (such as device model).
In our example we have named the Hub device site1-H1.
3. Select Add it to the Device Group, and select Hubs.
4. Select Assign the Policy Package, and select Hubs.

5. Select Assign Provisioning Template, and select default.
6. Click Next, and complete the wizard.

Configuring Meta Fields
To configure Meta Fields:
1. Right-click the Model Device named Hub, and select Edit.
2. Fill in following Meta Fields used in CLI templates for the Hub device.
The following values correspond to our example:
Meta Field site1-H
as 65001
inet-id 11
inet-intf port1
inet-tunnel-net 10.201.1.0/24
lan-net 10.1.0.0/24
lan-summary 10.0.0.0/14
mpls-id 12
mpls-intf port4

Meta Field site1-H
mpls-tunnel-net 10.202.1.0/24
tunnel-mask 255.255.255.0
3. Optionally, set device location and/or other desired parameters.
After you complete the Meta Fields, it's a good time to add any other required configuration to
the Model Device. It can be done either directly in the Device Manager or using Provisioning
Templates, additional CLI Templates, or ad-hoc CLI Scripts.
One typical example is underlay configuration. You may need to configure the missing
VLAN interfaces, IP addresses, static routes, dynamic routing on the underlay and so on. This
configuration is not specific to SD-WAN, and therefore it is out of scope for this document.
Generating and issuing certificates for Hub devices
To generate and issue certificates for Hub devices:
1. Navigate to Provisioning Templates > Certificate Templates.

2. Right-click on Hub template, and select Generate.

3. Select the Hub Model Device in the subsequent dialog, and click OK to issue a local certificate for the Hub device.
Installing the overlay configuration
Now it is necessary to install the overlay configuration on the Model Device. We must perform this step separately,
because other templates will require the overlay interfaces to exist.
To install the overlay configuration:
1. Go to Device Manager > Device & Groups.

2. From the Column Settings menu, select CLI Template Status.
The CLI Template Status column is displayed.
3. For the Hub device, right-click the CLI Template Status cell, and go to Assign CLI Template to select only the CLI
template named 01-Hub-Overlay:

4. Right-click the Hub device, and select Quick Install (Device DB) to install the 01-Hub-Overlay policy on the device:

Installing the remaining configuration to the model device
After successful configuration of the overlays, we are ready to install the rest of the configuration.
To install the remaining configuration to the model device:
1. Go to the Device Manager > Device & Groups pane, and locate the Hub device.
2. For the Hub device, right-click the CLI Template Status cell, and select the CLI Template Group named Hub-
Template.
The CLI Template Group is assigned to the Hub device.
3. Go to the SD-WAN > SD-WAN Templates.
4. Select the Hub-Template, and click Assign to Device.
The Assign to Device dialog box is displayed.

5. In the Available Entries list, select the Hub device, and click the right arrow (>) to move it to the Selected Entries list,
and click OK.
The template is assigned to the Hub device.
6. Right-click the Hub device, and select Quick Install (Device DB) to install the policies to the Hub device.
All of the assigned templates are installed. After successful installation, the Hub Model Device is ready:
Connecting the real device
Now it is time to connect the real device. You can use either Zero-Touch Provisioning, or you can manually initiate the
registration from FortiGate CLI.

Once the real device is connected and online, the Hub will become a fully managed and a completely deployed device:
Configuring Edge devices
Following is an overview of configuring edge devices:

1. Configure overlays. See Configuring overlay (Edge) on page 51.
2. Configure routing. See Configuring routing (Edge) on page 55.
3. Configure SD-WAN. See Configuring SD-WAN (Edge) on page 58.
4. Configure the firewall policy. See Configuring firewall policies (Edge) on page 67.
5. Deploy edge devices. See Deploying Edge devices on page 70.
Configuring overlay (Edge)
This section leverages CLI Templates once again to define the IPsec tunnels that will function as our overlay. Four new
Meta Fields are created to be used with two previously defined Meta Fields.
Following is a summary of the tasks required to configure the overlay:
1. In Device Manager, create a new CLI template to configure IPsec tunnels. See Creating a CLI template on page 52.
2. Add the template to a CLI template group. See Adding the template to a CLI template group on page 54.
3. Define four new Meta Fields. See Defining Meta Fields on page 54.
Previously defined Meta Fields will be used in the CLI template as well.

In this section, you'll create a CLI template called 01-Edge-Overlay in the Device Manager module. For more information
on how to create CLI Templates, see the FortiManager Administration Guide > CLI Templates chapter.
This CLI template will configure static IPSEC overlay tunnels that the Edge devices will establish to the Hub, over each of
the underlay transports. Most of the parameters used in Edge configuration must correspond to the Dial-Up IPSEC
endpoints that we have configured on the Hub. You should pay attention to the following guidelines:
l Create a separate static IPSEC tunnel to the Hub over each underlay transport
l Use IKEv2 with Mode Config, in order to automatically receive the tunnel IP addresses from the Hub
l Enable certificate-based authentication (use “Edge” certificate, matching the name of the Certificate Template
created earlier)
l Disable add-route feature, since the routing will be handled by BGP
l Use net-device enable mode (this is the only mode supported for ADVPN shortcut monitoring)
l Enable ADVPN receiver
l Enable network-overlay feature and set the Network ID to match the one set on the Hub
l Set the right remote-gw IP address, using the corresponding underlay IP addresses of the Hub
Following is the content of this CLI Template in our example topology, and it follows all the above guidelines:
edit "H1_INET"
set ike-version 2
set keylife 28800
set peertype any
set net-device enable
set mode-cfg enable
set idle-timeout enable
set auto-discovery-receiver enable
set network-id $(h1-inet-id)
set remote-gw $(h1-inet-ip)
set certificate "Edge"
set dpd-retrycount 3
set dpd-retryinterval 5
set dpd on-idle
next
edit "H1_MPLS"
set ike-version 2
set keylife 28800
set peertype any
set mode-cfg enable
set network-id $(h1-mpls-id)

set remote-gw $(h1-mpls-ip)

set dpd on-idle
next
end
edit "H1_INET"
set phase1name "H1_INET"
set keylifeseconds 3600
next
edit "H1_MPLS"
set phase1name "H1_MPLS"
next
end
edit "H1_INET"
set vdom "root"
set type tunnel
next
edit "H1_MPLS"
set vdom "root"
set type tunnel
next
end
h1-inet-id Network ID for the Internet overlay, as defined on the Hub
h1-inet-ip Underlay (WAN) IP of the Internet link on the Hub
h1-mpls-id Network ID for the MPLS overlay, as defined on the Hub
h1-mpls-ip Underlay (WAN) IP of the MPLS link on the Hub
*You may have already created the inet-intf and mpls-intf Meta Fields. If this is the case, you do not need to create them
again.

Note that the values of these Meta Fields will be identical for all Edge devices connecting to
the same Hub. Hence, if the goal is to build just a single Hub-and-Spoke topology, there is no
need to define these Meta Fields. The right values can be simply “hard-coded” into the CLI
templates. However, we will guide you to build a more generic and reusable set of templates.

2. From the Create New menu, select CLI Template, and create a new template called 01-Edge-Overlay.
Adding the template to a CLI template group
In this section, you'll create a CLI Template Group called Edge-Template, and add the overlay CLI template to the CLI
Template Group.
To add the template to the CLI template group:

2. From the Create New menu, select CLI Template Group, and create a new CLI Template Group called Edge-
Template.
3. Add the 01-Edge-Overlay CLI template to the group.


h1-inet-id Network ID for the Internet overlay, as defined on the Hub
h1-inet-ip Underlay (WAN) IP of the Internet link on the Hub
h1-mpls-id Network ID for the MPLS overlay, as defined on the Hub
h1-mpls-ip Underlay (WAN) IP of the MPLS link on the Hub
Configuring routing (Edge)
A CLI Template is used to complete the BGP configuration on the Edge device. Two routes are added to account for
when the tunnels are down and for inter edge communication over dissimilar overlays.
Following is a summary of the steps required to configure routing:
1. In Device Manager, create a CLI template for BGP configuration on edge devices. See Creating a CLI template on
page 55.
2. Add the CLI template to the existing CLI template group. See Adding the CLI template to the existing template group
on page 57.
3. Define the Meta Fields for use in CLI templates. See Defining Meta Fields on page 57.
In Device Manager, create a CLI template called 02-Edge-Routing, and add it to the CLI Template Group called Edge-
Template.
This template will configure IBGP sessions to the Hub, over each of the IPSEC overlays. You should follow the following
guidelines:
l Enable IBGP Multipath and ADD-PATH features
l Advertise a LAN subnet(s) of the local site behind the Edge device
In addition to the BGP configuration, it is important to configure the following static routes on Edge devices:
l A blackhole route to the summary of LAN prefixes of all SD-WAN sites within the region. In the simplest case (with
a single region), it can simply be the entire corporate network summary (such as an RFC1918 network). This
blackhole route not only helps avoiding unintentional leak of the corporate traffic to the underlay (for example, when
all the overlay tunnels are down), but also is mandatory for the correct operation of SD-WAN switchover in
conjunction with ADVPN shortcuts.
l A tunnel summary route (summarizing all overlay tunnel subnets). It allows cross-overlay communication, when
two Edge devices are not connected to a common overlay.
For example, if one Edge device is currently connected only to H1_INET overlay, while
another Edge device - only to H1_MPLS overlay, they will use these static routes in order
to resolve the reflected BGP next-hop IPs of each other.
Here is the content of this CLI Template in our example topology, that follows all the above guidelines:


edit 101
set dst $(lan-summary)
next
edit 102
set dst 10.200.0.0/14
set device "H1_INET"
set comment "Cross-overlay BGP NH reachability"
next
edit 103
set dst 10.200.0.0/14
set device "H1_MPLS"
next
end
config router bgp

set as $(as)
config neighbor
edit $(h1-inet-tunnel-ip)
set interface "H1_INET"
set connect-timer 1
set remote-as $(as)
set additional-path receive
next
edit $(h1-mpls-tunnel-ip)
set interface "H1_MPLS"
set connect-timer 1
set remote-as $(as)
next
end
config network
edit 1
next
end
end
Note that the tunnel summary (10.200.0.0/14) is “hard-coded” into the template for simplicity. You may need to review it
and adjust to your environment. Alternatively, it could be defined as additional Meta Field.

Value Description
lan-net LAN subnet behind the Edge
h1-inet-tunnel-ip Tunnel IP of the Internet overlay on the Hub
h1-mpls-tunnel-ip Tunnel IP of the MPLS overlay on the Hub
*You may have already created some Meta Fields. If this is the case, you do not need to create them again.

2. From the Create New menu, select CLI Template, and create a new template called 02-Edge-Routing.
Add the CLI template called 02-Edge-Routing to the CLI Template Group called Edge-Template.
To add the CLI template to the existing CLI template group:

2. Under CLI Template Group, double-click the Edge-Template to open it for editing.
3. Beside Members, click + to select the 02-Edge-Routing CLI Template, and add it to the group.
Define Meta Fields for use in CLI templates.


For all Meta Fields, set Object to Device, and set Importance to Optional. Note that some of them have been already
created when configuring the Hub, and we are reusing them.
Value Description
lan-net LAN subnet behind the Edge
h1-inet-tunnel-ip Tunnel IP of the Internet overlay on the Hub
h1-mpls-tunnel-ip Tunnel IP of the MPLS overlay on the Hub
Configuring SD-WAN (Edge)
While overlay and routing configuration are generic, the SD-WAN configuration depends on the traffic steering strategy
that you would like to implement. In our example, in addition to the legacy MPLS service interconnecting our sites, we
are willing to utilize a more economical broadband Internet connection, as long as it is able to meet the SLA targets for
our business-critical applications. We will demonstrate the following policy:
l All corporate traffic (Edge-to-Edge and Edge-to-Hub) must prefer an overlay over Internet, as long as it meets the
SLA target. If it doesn’t, then the traffic must switchover to an overlay over MPLS.
l Edge-to-Hub traffic will flow via the static overlay tunnels towards the Hub, while Edge-to-Edge traffic will flow via
direct tunnels dynamically built by ADVPN whenever needed (“shortcuts”).
l Edge devices will implement Direct Internet Access (DIA), preferring it for all Edge-to-Internet traffic, as long as it
meets the SLA target. If it doesn’t, then the traffic must switchover to an overlay over MPLS for Remote Internet
Access (RIA) via the Hub.
This policy is presented here for example purposes only. The SD-WAN rules can be adjusted
to your specific needs, with rich feature set available, including Application-aware steering,
integration with User Identity services, different link selection strategies, WAN Path
Remediation tools and more.
Following is a summary of how to configure the SD-WAN for edge devices:

1. Define interface members. See Defining interface members (Edge) on page 59.
2. Configure health-check servers. See Configuring health-check servers (Edge) on page 60.
3. Configure SD-WAN template. See Creating an SD-WAN template (Edge) on page 60.
4. Configure SD-WAN as the default route. See Configuring SD-WAN as default route (Edge) on page 67.

Defining interface members (Edge)
Similar to the hub, we will define two SD-WAN interfaces for the overlay VPNs. In this section, we'll create two interface
members for the internet and MPLS overlays, and map them to the IPsec tunnels on the edge.
In Device Manager, navigate to centralized SD-WAN management tab and start by defining Interface Members.
Remember to set the “priority” value of 10 for both overlays!
To define interface members:
1. Go to Device Manager > SD-WAN > Interface Members.

2. Create a new interface member.
a. Click Create New.

b. In the Name box, type H1_INET.
c. In the Normalized Interface list, select the respective IPSEC tunnel on the Edge.
d. Under Advanced Options, set priority value to 10.
We give higher priority value (meaning lower priority) to all the overlay interfaces, so that Internet traffic uses
underlay interfaces by default. This includes all the locally originated Internet traffic (such as FortiGate
communication with FortiGuard services), as well as all the Internet traffic that is not explicitly controlled by SD-
WAN rules.
e. Click OK.
The interface is saved.
f. Repeat this procedure to create an interface member named H1_MPLS.
We will reuse the UL_INET Interface Member created for the Hub in this example.

Configuring health-check servers (Edge)
Health check servers are destinations used by SD-WAN probes to measure link quality. They can be internet services
(for example, www.fortinet.com) or internal servers (such as a server on your corporate LAN subnet).
Within the SD-WAN section, we will configure the following health-check servers:
l A health-check server for the Internet
l A health-check server for the hub
Health-Check Servers are the destinations for the health probes sent by SD-WAN to monitor link quality.
To configure health-check servers:
1. In FortiManager, go to Device Manager > SD-WAN > Health-Check Servers.

2. Click Create New, and configure a health-check server called Internet that probes an Internet destination, such as
www.fortinet.com.
3. Create another health-check server named HUB with the IP of the loopback interface lo-HC that we have configured
on the Hub.
Creating an SD-WAN template (Edge)
SD-WAN template is how you input the settings which will govern how the traffic is sent through the use of SLAs, health
checks and rules.
Following is a summary of how to configure SD-WAN template:

1. In Device Manager, create a new SD-WAN template for Edge devices. See Creating an SD-WAN template for edge
devices on page 61.
2. Create two SD-WAN zones named overlay and underlay. See Creating SD-WAN zones on page 61.
3. Create three SD-WAN interface members, and put them in their respective underlay or overlay zone. See Creating
SD-WAN interface members on page 61.
4. Create the following performance SLAs:
a. Internet
b. Hub loopback interfaces over each overlay
See Creating performance SLA on page 62.
5. Configure two SD-WAN rules to steer traffic. See Configuring SD-WAN rules to steer traffic on page 65.
Creating an SD-WAN template for edge devices
To create a template for edge devices:
1. In FortiManager, go to Device Manager > SD-WAN > SD-WAN Templates, and click Create New.
2. In the Name box, type Edge-Template.
3. Go to the next procedure to create SD-WAN Zones.
Creating SD-WAN zones
Create two SD-WAN Zones named overlay and underlay in the SD-WAN template named Edge-Template. Keep them
empty at the moment.
To create SD-WAN zones:
1. In the SD-WAN template under Interface Members, click Create New > SD-WAN Zone.
2. In the Name box type overlay, and click OK.
The SD-WAN zone is created.
3. Repeat this procedure to create an SD-WAN zone named underlay.
4. Go to the next procedure to create SD-WAN interface members.
Creating SD-WAN interface members
Create SD-WAN interface members for this template, and assign them to the appropriate SD-WAN Zones, as follows:
UL_INET underlay
H1_INET overlay
H1_MPLS overlay
To create SD-WAN interface members:
1. In the SD-WAN template under Interface Members, click Create New > SD-WAN Member.
The Create New SD-WAN Interface Member dialog box is displayed.
2. In the Interface Member list, select an interface member.

3. In the SD-WAN Zone list, select the zone, and click OK.
The interface member is created.
4. Repeat this procedure until you create all SD-WAN members.
5. Go to the next procedure to create performance SLA.
Creating performance SLA
To create performance SLA:
1. Create Performance SLAs defining the desired SLA targets and health checks for each Interface Member.
Create one Performance SLA to be used for the corporate traffic, probing the lo-HC interface of the Hubs over
members H1_INET and H1_MPLS.
a. Under Performance SLA, click Create New.
The Create New Performance SLA dialog box is displayed.
b. In the Name box, type HUB.
c. In the Health-Check Server list, select HUB.
d. Beside Participants, select Specify, and then select H1_INET and H1_MPLS.
e. Under SLA, click Create New.
f. Beside Latency type 100, leave the default for Jitter Threshold, and change Packet Loss Threshold to 10, and
click OK.
The SLA is created.

2. Create Performance SLAs defining the desired SLA targets and health checks for each Interface Member.
l Create one Performance SLA to be used for the corporate traffic, probing the lo-HC interface of the Hub over
members H1_INET and H1_MPLS:

l Create another Performance SLA to be used for the Internet traffic, probing the Internet health check over
members UL_INET and H1_MPLS
In each Performance SLA, configure the non-zero values of the sla-fail-log-period

and sla-pass-log-period parameters under the Advanced Options. These values
define the time interval for sending status updates to FortiAnalyzer. Hence, they have
impact on the accuracy of graphs and reports.

Configuring SD-WAN rules to steer traffic
To configure SD-WAN rules to steer traffic:
1. Configure SD-WAN Rules that will consolidate all the elements together, in order to steer the traffic. In this example,
we will configure the following two SD-WAN Rules:
l A rule for the corporate traffic that will prefer H1_INET overlay, as long as it meets the SLA target. If it doesn’t,
then H1_MPLS overlay will be used instead:
Note that this single SD-WAN rule will cover both Edge-to-Hub and Edge-to-Edge traffic.
l Edge-to-Hub traffic will simply flow via the overlay selected by the SD-WAN rule
l For Edge-to-Edge traffic, direct IPSEC tunnel (ADVPN shortcut) will be dynamically built between the two
Edge devices, using the overlay selected by the SD-WAN rule.

For SD-WAN Rules applying “Lowest Cost (SLA)” strategy over ADVPN shortcuts, we highly recommend
configuring hold-down-time parameter under the rule’s Advanced Options. Setting its value to 20 seconds (or
more) will ensure correct switchover behavior from secondary shortcut back to the primary (after the network
recovers from a failure). Without setting this parameter, the traffic may switchover prematurely, while the
network issue still persists, causing a short period of bad application quality.
l A rule for the Internet traffic that will prefer Direct Internet Access (DIA), as long as the underlay connection
(UL_INET) meets the SLA target. If it doesn’t, then H1_MPLS overlay will be used instead (so that Internet
traffic will be backhauled via the Hub):

This SD-WAN configuration does not cover Hub-to-Edge traffic (sessions originated behind
the Hub). As a result, if such traffic exists, it will be handled by the traditional routing,
disregarding the health of the overlays towards the Edge devices. If it is desired to let SD-WAN
select the optimal path for Hub-to-Edge traffic, please follow the steps detailed in Signaling
SLA status to hubs on page 114.
Configuring SD-WAN as default route (Edge)
Just as with the hub, it is recommended that SD-WAN act as the default route. This will automatically generate FortiOS
default route via all the configured SD-WAN Interface Members.
We will reuse the same CLI template 03-Default that we used on the Hub and add it to the CLI template group Edge-
Template.
As already mentioned, configuring SD-WAN to act as a default route is not mandatory, but it
ensures that SD-WAN rules are fully responsible for traffic steering, making the routing
configuration generic. For more details, see SD-WAN routing logic on page 122.
To add the template to the existing template group:

2. Under CLI Template Group, double-click the Edge-Template to open it for editing.
3. Beside Members, click + to select 03-Default, and add it to the CLI template group named Edge-Template.
Configuring firewall policies (Edge)
Just like on the hub, the firewall policies control and secure what traffic is permitted, and from which source and to which
destination.
Following is a summary of how to configure firewall policies:

1. Create a new policy package for Edge devices and assign it to the Edge group. See Creating a new policy package
on page 68.
2. Configure 3 rules for the edge devices. See Creating rules on page 69.
Creating a new policy package
To create a new policy package:
1. In Policy & Objects, create a new Policy Package called Edge, and add the Device Group Edge to its installation
targets:

Creating rules
To create rules
1. Create the following Firewall Policy:
Corporate lan overlay lan overlay CORP_ CORP_ ALL No Accept

LAN LAN
Internet lan underlay all all ALL Yes Accept

(DIA)
Internet lan overlay all all ALL No Accept

(RIA)
Notes:
l The SD-WAN Zones underlay and overlay were automatically created to be used in the Firewall Policy
l The Firewall Policy distinguishes between Direct Internet Access (DIA, from the Edge itself) and Remote
Internet Access (RIA, via the Hub), potentially applying different security features in each case. One common
example shown above is Source NAT which is only applied to the traffic using DIA.
l It is highly recommended to enable Application Control, especially on the Firewall Policy controlling Internet
traffic. For accurate application identification, it is also highly recommended to enable SSL Inspection. This
functionality is required for SD-WAN Application-aware traffic steering and is also used to populate SD-WAN
widgets and reports.

Deploying Edge devices
Now all the building blocks are ready, and it is time to deploy the Edge. Following is an overview of deploying Edge
devices:
1. Add a Model Device. See Adding model devices on page 71.
a. Ensure there is a thoughtful naming convention.
b. Add the model device to the Edge device group.
c. Assign the Edge policy package.
d. Assign the default provisioning template.
2. Define the Meta Fields. See Defining Meta Fields on page 73.
3. Generate a certificate using the certificate template. See Generating and assigning certificates on page 75.
4. Assign and install the edge overlay CLI template to the Model Device. See Installing overlay configuration on page
75.
5. Assign the remaining templates to the Model Device. See .
l CLI Template group
l SD-WAN template
l Policy package
6. Install the templates to the Model Device. See Installing the templates to the model device on page 75.
7. Onboard the real FortiGate. See Onboarding the real device on page 76.

Adding model devices
To add model device:
1. On Device Manager, click Add Device, and select Add Model Device.
2. Give it a name of the Edge, and select the right parameters (such as device model).
In our example, we have named Edges site1-1 or site1-2.
3. Select Add it to the Device Group, and select Edge.
4. Select Assign the Policy Package, and select Edge.

5. Select Assign the Provisioning Template, and select default.
6. Click Next, and complete the wizard.

1. Right-click the Hub Model Device, and select Edit.
2. Fill in following Meta Fields used in Edge CLI templates.

The values correspond to our example, for both Edge devices.
Meta Field site1-1 site1-2
as 65001 65001

h1-inet-id 11 11
h1-inet-ip 100.64.1.1 100.64.1.1
h1-inet-tunnel-ip 10.201.1.1 10.201.1.1
h1-mpls-id 12 12
h1-mpls-ip 172.16.1.5 172.16.1.5
h1-mpls-tunnel-ip 10.202.1.1 10.202.1.1
inet-intf port1 port1
lan-net 10.0.1.0/24 10.0.2.0/24
lan-summary 10.0.0.0/14 10.0.0.0/14
mpls-intf port4 port4
In the above list, only lan-net Meta Field is unique per Edge device (site). All other values
are determined by the region to which a particular site belongs. As a result, they will be
identical on all the Edge devices in the same region.
3. Optionally, set device location and/or other desired parameters.
After you complete Meta Fields, it is a good time to add any other required configuration to
the Model Device. It can be done either directly in the Device Manager or using
Provisioning Templates, additional CLI Templates, or ad-hoc CLI Scripts.
One typical example is underlay configuration. You may need to configure the missing
VLAN interfaces, IP addresses, static routes, dynamic routing on the underlay and so on.
This configuration is not specific to SD-WAN, and therefore it is out of scope for this
document.

Generating and assigning certificates
To generate and issue certificates:
1. Navigate to Provisioning Templates > Certificate Templates.

2. Right-click the Edge template, and select Generate.
3. Select the Edge Model Device in the subsequent dialog, and click OK to issue a local certificate for the Edge.
Installing overlay configuration
Now it is necessary to install the overlay configuration on the Model Device. Just as for the Hub, we must perform this
step separately, before applying the rest of the templates.
1. For the Edge device, right-click the CLI Template Status cell, and select only the CLI template named 01-Edge-
Overlay.
2. Install the configuration by using the Quick Install (Device DB) method.
Assigning templates to the model device
After successful configuration of the overlays, we are ready to assign the rest of the templates to the Model Device.
To assign templates to the model device:
1. For the Edge, right-click the CLI Template Status cell, and select the entire CLI Template Group named Edge-
Template.
2. Navigate to SD-WAN tab, and assign the SD-WAN template, for example, Edge-Template.
Installing the templates to the model device
After successful configuration of the overlays, we are ready to install the rest of the configuration.
To install the rest of the configuration:
1. Install Edge policy on the Edge Model Device, which will also apply all the assigned templates.

Onboarding the real device
After successful installation, the Edge Model Device is ready. Now it is time to onboard the real device. You can use
either Zero-Touch Provisioning, or you can manually initiate the registration from FortiGate CLI.
Once the onboarding process is complete, the Edge will become a fully managed and a completely deployed device.
Repeat this procedure for all Edge devices. Note that we reuse the same CLI templates and the same SD-WAN
templates, and only the Meta Fields must be filled in for each device.
Verifying configuration
At this point, your SD-WAN/ADVPN topology should be fully deployed.

In Device Manager, go to SD-WAN > Monitor, and confirm that all the Edge devices report the health of all their SD-WAN
Members:
You can drill down by clicking on Edge device:

If all the overlays are healthy, this also implies that the IPSEC tunnels are up. You can confirm this on the centralized
view under VPN Manager > Monitor:
For a more granular per-device view, you can use multiple widgets available for each managed device in the Device
Manager:

Make sure the Edge device learns BGP routes corresponding to the LAN networks behind remote Edge device(s) and
behind the Hub itself.
Here you can also confirm different aspects of your device configuration. For example, go to System: Certificates section
of one of the Edge devices, and confirm that FMG has pushed the issued local certificate (“Edge”) and also added its
own CA certificate to the trusted list:

Finally, the following CLI commands will be useful for a more advanced troubleshooting. The outputs are shown for
reference, from one of the Edge devices in our example:
Overlay:
site1-1 # get ipsec tunnel list
NAME REMOTE-GW PROXY-ID-SOURCE PROXY-ID-DESTINATION STATUS TIMEOUT
H1_INET 100.64.1.1:0 0.0.0.0/0.0.0.0 0.0.0.0/0.0.0.0 up 3174
H1_MPLS 172.16.1.5:0 0.0.0.0/0.0.0.0 0.0.0.0/0.0.0.0 up 2795
Routing:
site1-1 # get router info bgp summary
VRF 0 BGP router identifier 10.0.1.1, local AS number 65001
BGP table version is 3
1 BGP AS-PATH entries
0 BGP community entries
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd

10.201.1.1 4 65001 1520 1517 2 0 0 01:50:44 4
10.202.1.1 4 65001 1606 1600 1 0 0 01:57:09 4
Total number of neighbors 2

site1-1 # get router info routing-table bgp
Routing table for VRF=0
B 10.0.2.0/24 [200/0] via 10.201.1.3, H1_INET, 01:50:02 [200/0] via 10.202.1.3, H1_MPLS,
01:50:02 [200/0] via 10.201.1.3, H1_INET, 01:50:02 [200/0] via 10.202.1.3, H1_MPLS,
01:50:02
01:50:02
SD-WAN:
site1-1 # diagnose sys sdwan health-check
Health Check(HUB):
Seq(2 H1_INET): state(alive), packet-loss(0.000%) latency(1.542), jitter(0.194) sla_map=0x1
Seq(3 H1_MPLS): state(alive), packet-loss(0.000%) latency(1.122), jitter(0.176) sla_map=0x1
Health Check(Internet):
Seq(1 port1): state(alive), packet-loss(0.000%) latency(12.012), jitter(1.068) sla_map=0x1

Dual Hub deployment
Dual Hub deployment
Building on the previous section, the second hub is added in the same way the first was. In this chapter, we will extend
the basic configuration to include two Hubs operating as Primary/Secondary.
2. Complete a Single Hub deployment, which includes Meta Fields and Normalized Interfaces. See Single Hub
deployment on page 24.
Following is a summary of how to configure Dual Hub SD-WAN:
1. Configure hub devices. See Configuring Hub devices on page 80.
2. Configure edge devices. See Configuring Edge devices on page 81.
See also Dual Hub topology on page 9.
Configuring Hub devices
Hub configuration remains unchanged. We will reuse the templates created in the previous chapter. See Configuring the
Hub device on page 24.
To onboard the Secondary Hub, follow the instructions in Deploying the Hub device on page 43. We will call it site1-H2.
Make sure you fill in the right values for its Meta Fields. The Meta Field values should, of course, differ from those set for
the Primary Hub. The following values are used in our example:
Meta Field site1-H1 site1-H2
as 65001 65001
inet-id 11 21
inet-tunnel-net 10.201.1.0/24 10.201.2.0/24
mpls-id 12 22
mpls-tunnel-net 10.202.1.0/24 10.202.2.0/24
tunnel-mask 255.255.255.0 255.255.255.0
lan-summary 10.0.0.0/14 10.0.0.0/14
lan-net 10.1.0.0/24 10.2.0.0/24
Finish the onboarding process. You now have two fully managed and completely deployed Hub devices:

Dual Hub deployment
Following is a summary of how to configure Edge devices for dual Hub SD-WAN:
1. Configure the overlay. See Configuring overlay on page 81.
2. Configure routing. See Configuring routing on page 85.
3. Configure SD-WAN template. See Configuring SD-WAN template on page 87.
4. Configure firewall policy. See Configuring firewall policies on page 92.
5. Deploy Edge devices. See Deploying Edge devices on page 92.
Configuring overlay
Here we double the existing configuration and create two VPN tunnels from each underlay to each hub, resulting in four
tunnels total.
Following is a summary of how to configure the overlay:
1. Edit the existing overlay template, or create a new overlay CLI template. See Creating an overlay template on page
81.
With 2 hubs, each underlay will establish 2 VPN tunnels from each WAN connection.
2. Create the four new Meta Fields in FortiManager for the new tunnels. See Creating Meta Fields on page 84.
3. Create a new template group and add this template (or if reusing, skip this step). See Creating a template group on
page 84.
Creating an overlay template
Create (or import) a new CLI template called 01-Edge-DualHub-Overlay.conf.

As in the basic single-Hub configuration, this CLI template will configure static IPSEC overlay tunnels that the Edge
devices will establish to the Hubs - this time to both of them. We will follow the same guidelines described in the previous
chapter.
Here is the content of this CLI Template in our example topology:
edit "H1_INET"
set ike-version 2
set keylife 28800
set peertype any

Dual Hub deployment
set mode-cfg enable

set dpd on-idle
next
edit "H1_MPLS"
set ike-version 2
set keylife 28800
set peertype any
set mode-cfg enable
set dpd on-idle
next
edit "H2_INET"
set ike-version 2
set keylife 28800
set peertype any
set mode-cfg enable
set dpd on-idle
next
edit "H2_MPLS"
set ike-version 2

Dual Hub deployment
set keylife 28800

set peertype any
set mode-cfg enable
set dpd on-idle
next
end
edit "H1_INET"
next
edit "H1_MPLS"
next
edit "H2_INET"
next
edit "H2_MPLS"
next
end
edit "H1_INET"
set vdom "root"
set type tunnel
next
edit "H1_MPLS"
set vdom "root"
set type tunnel
next
edit "H2_INET"

Dual Hub deployment
set vdom "root"

set type tunnel
next
edit "H2_MPLS"
set vdom "root"
set type tunnel
next
end
Creating Meta Fields
You will notice the following additional Meta Fields used:
h2-inet-id Network ID for the Internet overlay, as defined on the Secondary Hub
h2-inet-ip Underlay (WAN) IP of the Internet link on the Secondary Hub
h2-mpls-id Network ID for the MPLS overlay, as defined on the Secondary Hub
h2-mpls-ip Underlay (WAN) IP of the MPLS link on the Secondary Hub
To create Meta Fields:
1. Navigate to System Settings > Advanced > Meta Fields, and create the missing Meta Fields listed in the table
above. All the Meta Fields must be of type Device and defined as Optional.
Creating a template group
To create a new CLI Template Group:
1. Create a new CLI Template Group called Edge-DualHub-Template, and add the overlay CLI template to it.

Dual Hub deployment
Configuring routing
Similar to single hub, we need iBGP sessions to each to propagate routes.

Following is a summary of how to configure routing:
1. Edit the existing overlay template, or create a new overlay CLI template. See Creating a template on page 85.
2. Define new Meta Fields for the second Hub. See Defining Meta Fields on page 86.
3. Add the template to the template group. See Adding the CLI template to the group on page 86.
Creating a template
Create (or import) another CLI Template, called 02-Edge-DualHub-Routing.

As before, this template will configure IBGP sessions to the Hubs - this time, to both of them. We will follow the same
guidelines described in the previous chapter.
Here is the content of this CLI Template in our example topology:
edit 101
set dst $(lan-summary)
next
edit 102
set dst 10.200.0.0/14
set device "H1_INET"
next
edit 103
set dst 10.200.0.0/14
set device "H1_MPLS"
next
end
config router bgp

set as $(as)
config neighbor
set connect-timer 1
set remote-as $(as)
next

Dual Hub deployment
set connect-timer 1
set remote-as $(as)
next
set connect-timer 1
set remote-as $(as)
next
set connect-timer 1
set remote-as $(as)
next
end
config network
edit 1
next
end
end
You will notice the following additional Meta Fields used:
h2-inet-tunnel-ip Tunnel IP of the Internet overlay on the Secondary Hub
h2-mpls-tunnel-ip Tunnel IP of the MPLS overlay on the Secondary Hub
Navigate to System Settings > Advanced > Meta Fields, and create the missing Meta Fields listed in the table above. All
the Meta Fields must be of type Device and defined as Optional.
Adding the CLI template to the group
To add the CLI template to the group:
Add the new CLI template to the group Edge-DualHub-Template created earlier.

Dual Hub deployment
Configuring SD-WAN template
As in the previous chapter, we will demonstrate an example policy that can be adjusted to your specific needs:
l The Hubs implement Primary/Backup model. As long as Primary Hub is in service, it must be used. When Primary
Hub is unavailable, Secondary Hub must be used.
l The rest of the policy is identical to the one implemented in the previous chapter.
SD-WAN template also covers:
l Defining interface members on page 87
l Creating an SD-WAN template on page 87
l Configuring SD-WAN as default route on page 91
Defining interface members
From Device Manager, go to SD-WAN > Interface Members. Create two additional Interface Members for the Secondary
Hub overlays, H2_INET and H2_MPLS. The procedure is similar to the one we followed before.
Remember to set the priority value of 10 for both overlays!
We will reuse all the rest of the Interface Members.
Creating an SD-WAN template
The SD-WAN template defines the SLA performance among other things.
Following is a summary of how to configure the template:
1. Create a new SD-WAN Template called Edge-DualHub-Template. See Creating an SD-WAN template on page 87.
2. Create two SD-WAN zones. See Configuring SD-WAN zones on page 87.
3. Create SD-WAN members. See Configuring SD-WAN members on page 88.
4. Create performance SLA. See Configuring performance SLA on page 88.
5. Configure SD-WAN rules. See Configuring SD-WAN rules on page 89.
Creating an SD-WAN template
To create an SD-WAN template:
1. Create a new SD-WAN Template called Edge-DualHub-Template.
Configuring SD-WAN zones
To configure SD-WAN zones:
1. Create two SD-WAN Zones called overlay and underlay, and keep them empty at the moment.

Dual Hub deployment
Configuring SD-WAN members
To configure SD-WAN members:
1. Create SD-WAN Members for this template, assigning them to the appropriate SD-WAN Zones, as follows:
UL_INET underlay
H1_INET overlay
H1_MPLS overlay
H2_INET overlay
H2_MPLS overlay
Configuring performance SLA
Create Performance SLAs that define the desired SLA targets and health checks for each Interface Member.
l Create one Performance SLA to be used for the corporate traffic, probing the lo-HC interface of the Hubs. Note that
we reuse the same loopback IP address on both Hubs.

Dual Hub deployment
l Create another Performance SLA to be used for the Internet traffic, probing the Internet health check (unchanged
from previous chapter).
Remember to set the sla-fail-log-period and sla-pass-log-period parameters under the Advanced Options.
Configuring SD-WAN rules
Finally, configure SD-WAN Rules. For Primary/Secondary Hub behavior, we recommend configuring two rules, as
follows:
l The first rule (Corpoarte-H1) will include only H1_INET and H1_MPLS overlays, and will prefer the former, as long
as it meets the SLA target. This rule is identical to the Corporate rule from the previous chapter.

Dual Hub deployment
l The second rule (Corporate-H2) will include only H2_INET and H2_MPLS overlays, and will prefer the former, as
long as it meets the SLA target.
During SD-WAN rule lookup process, the first rule that matches will be used to forward traffic, unless all its members are
either unavailable or have no valid route to destination.
Let us consider the following two examples:
l Edge-to-Edge traffic. Since both Hubs reflect all Edge routes, there are valid routes to any remote Edge via both
Hubs. But once our traffic matches Corporate-H1 rule, the selection will be made only between H1_INET and H1_
MPLS. Secondary Hub overlays will not be considered, even if both Primary Hub overlays are out of SLA.
However, if the Primary Hub is down, none of its overlays will be available. In that case, the rule will be skipped, and
the traffic will match Corporate-H2 rule, selecting between H2_INET and H2_MPLS.

Dual Hub deployment
This is inline with the Primary/Secondary Hub model: the Secondary Hub will only be used, if the Primary Hub is
completely out of service.
l Edge-to-Hub traffic. Here the behavior will be different, because each Hub in our example advertises its own LAN
prefix to the Edge devices. Hence, even if the Primary Hub is operational, the traffic towards Secondary Hub’s LAN
network will successfully skip the Corporate-H1 rule, match Corporate-H2 rule, and select between H2_INET and
H2_MPLS.
This avoids the situation where the traffic would be blackholed to the wrong Hub, and it would be unable to forward it
to the destination.
l Add the third rule for the Internet traffic. This rule will be similar to the one described in the previous chapter, with
one small difference: it will also include the new H2_MPLS member.
Configuring SD-WAN as default route
Just as with the single hub topology, we prefer to let SD-WAN handle the traffic steering.
We will reuse the same CLI template 03-Default that we have used in the previous chapter. Add it to the CLI template
group Edge-DualHub-Template:
As already mentioned, configuring SD-WAN to act as a default route is not mandatory, but it
ensures that SD-WAN rules are fully responsible for traffic steering, making the routing
configuration generic. For more details, see SD-WAN routing logic on page 122.

Dual Hub deployment
Configuring firewall policies
The Policy Package Edge that we have created in the previous chapter uses SD-WAN Zones. Since we are reusing the
same SD-WAN Zone overlay, which now contains the tunnels towards both Hubs, the firewall rules remain valid for the
Dual Hub SD-WAN topology.
Hence, we do not need to alter them.
Deploying Edge devices
The procedure is similar to deploying the edge in a single hub topology, but with additional Meta Fields.
Following is a summary of how to deploy the edge:
1. Fill out the Meta Fields for the edge device(s).
l For new devices, this takes place on the model device
l For existing devices migrating from single hub to dual hub, edit the connected device
Pay attention to the following:

l Remember that you must fill the values of the Meta Fields corresponding to both Hubs, namely:
as 65001 65001
h1-inet-id 11 11
h1-inet-ip 100.64.1.1 100.64.1.1
h1-inet-tunnel-ip 10.201.1.1 10.201.1.1
h1-mpls-id 12 12
h1-mpls-ip 172.16.1.5 172.16.1.5
h1-mpls-tunnel-ip 10.202.1.1 10.202.1.1
h2-inet-id 21 21
h2-inet-ip 100.64.2.1 100.64.2.1
h2-inet-tunnel-ip 10.201.2.1 10.201.2.1
h2-mpls-id 22 22
h2-mpls-ip 172.16.2.5 172.16.2.5
h2-mpls-tunnel-ip 10.202.2.1 10.202.2.1
lan-summary 10.0.0.0/14 10.0.0.0/14
lan-net 10.0.1.0/24 10.0.2.0/24
l When installing overlay configuration, make sure you use the new CLI template 01-Edge-DualHub-Overlay.

Dual Hub deployment
l When assigning the complete CLI Template Group, make sure you assign the new Edge-DualHub-Template.
l When assigning the SD-WAN Template, make sure you assign the new Edge-DualHub-Template.
Repeat the onboarding process for all Edge devices.
Verification steps are similar to those described in the previous chapter.

Navigate to SD-WAN => Monitor and confirm that all the Edge devices report the health of all their SD-WAN Members,
now including the overlay tunnels towards the Secondary Hub:
Similarly, confirm that the Edge devices now establish IPSEC tunnels to both Hubs and also learn BGP routes from both
of them:

Dual Hub deployment
Finally, the following outputs are shown for reference, from one of the Edge devices in our example:
Overlay:
site1-1 # get ipsec tunnel list
NAME REMOTE-GW PROXY-ID-SOURCE PROXY-ID-DESTINATION STATUS TIMEOUT H1_INET 100.64.1.1:0
0.0.0.0/0.0.0.0 0.0.0.0/0.0.0.0 up 1370 H2_INET 100.64.2.1:0 0.0.0.0/0.0.0.0
0.0.0.0/0.0.0.0 up 2965 H1_MPLS 172.16.1.5:0 0.0.0.0/0.0.0.0 0.0.0.0/0.0.0.0 up 991
H2_MPLS 172.16.2.5:0 0.0.0.0/0.0.0.0 0.0.0.0/0.0.0.0 up 2962
Routing:
site1-1 # get router info bgp summary
VRF 0 BGP router identifier 10.0.1.1, local AS number 65001
BGP table version is 5
1 BGP AS-PATH entries
0 BGP community entries
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd

10.201.1.1 4 65001 1949 1948 2 0 0 02:22:14 4
10.201.2.1 4 65001 82 80 3 0 0 00:05:43 3
10.202.1.1 4 65001 2032 2030 1 0 0 02:28:39 4
10.202.2.1 4 65001 84 80 4 0 0 00:05:42 3
Total number of neighbors 4

site1-1 # get router info routing-table bgp
Routing table for VRF=0
00:05:42 [200/0] via 10.201.2.2, H2_INET, 00:05:42 [200/0] via 10.201.1.3, H1_INET,
00:05:42 [200/0] via 10.202.1.3, H1_MPLS, 00:05:42 [200/0] via 10.201.2.2, H2_INET,
00:05:42
02:22:16
00:05:44
SD-WAN:
site1-1 # diagnose sys sdwan health-check

Dual Hub deployment
Health Check(HUB):
Health Check(Internet):
Seq(1 port1): state(alive), packet-loss(0.000%) latency(12.951), jitter(1.766) sla_map=0x1

Multi-Regional deployment
In this section, we describe how to extend our topology to multiple regions. Although we describe how to extend the
topology to just two regions, you can use the same procedure to add additional regions to the deployment.
2. Complete a Single Hub deployment, which includes Meta Fields and Normalized Interfaces. See Single Hub
deployment on page 24.
Following is an overview of the tasks needed to deploy Multi-Regional SD-WAN:
1. Review the addressing scheme. See Addressing scheme on page 96.
2. Deploy individual regions. See Deploying individual regions on page 96.
3. Interconnect the regions. See Interconnecting regions on page 98.
4. Configure edge devices with SD-WAN. See Configuring Edge devices with SD-WAN on page 111 .
See also Multi-Regional topology on page 10.
Addressing scheme
Careful IP address planning can significantly simplify multi-regional deployment. Route summarization is a key to
successful horizontal scaling, and Regional Hubs will advertise the summaries of their regional LAN prefixes to each
other.
For this reason, it is important that all the LAN prefixes of each particular region can be summarized, and these
summaries should not overlap between different regions!
The following addressing scheme is used in our example (as can be seen on the topology diagram in Multi-Regional
topology on page 10):
Region Summary (lan-summary)
Region 1 10.0.0.0/14
Region 2 10.4.0.0/14
Deploying individual regions
Start by deploying each of your regions, as described in the previous chapters. As shown in our example, it can be a mix
of Single Hub and Dual Hub regions.
Pay attention to the following guidelines:

l When filling in the value of lan-summary Meta Field, use the LAN summary of the particular region.
l Later we will advertise these summaries between the regions, to provide inter-regional connectivity. This
advertisement will be performed by the Hubs. Hence, fill in the right value of lan-summary not only on the Edge
devices, but also on the Hubs!
l Use a unique BGP Autonomous System (AS) number per region.
The following tables summarize the Meta Field values for all the devices in our example:
Hub devices:
Meta Field site1-H1 site1-H2 site2-H1
as 65001 65001 65002
inet-id 11 21 41
inet-intf port1 port1 port1
inet-tunnel-net 10.201.1.0/24 10.201.2.0/24 10.201.4.0/24
mpls-id 12 22 42
mpls-intf port4 port4 port4
mpls-tunnel-net 10.202.1.0/24 10.202.2.0/24 10.202.4.0/24
tunnel-mask 255.255.255.0 255.255.255.0 255.255.255.0
lan-summary 10.0.0.0/14 10.0.0.0/14 10.4.0.0/14
lan-net 10.1.0.0/24 10.2.0.0/24 10.4.0.0/24
Edge devices:
Meta Field site1-1 site1-2 site2-1
as 65001 65001 65002
inet-intf port1 port1 port1
mpls-intf port4 port4 port4
h1-inet-id 11 11 41
h1-inet-ip 100.64.1.1 100.64.1.1 100.64.3.1
h1-inet-tunnel-ip 10.201.1.1 10.201.1.1 10.201.4.1
h1-mpls-id 12 12 42
h1-mpls-ip 172.16.1.5 172.16.1.5 172.16.3.5
h1-mpls-tunnel-ip 10.202.1.1 10.202.1.1 10.202.4.1
h2-inet-id 21 21 -
h2-inet-ip 100.64.2.1 100.64.2.1 -
h2-inet-tunnel-ip 10.201.2.1 10.201.2.1 -
h2-mpls-id 22 22 -

Meta Field site1-1 site1-2 site2-1
h2-mpls-ip 172.16.2.5 172.16.2.5 -
h2-mpls-tunnel-ip 10.202.2.1 10.202.2.1 -
lan-summary 10.0.0.0/14 10.0.0.0/14 10.4.0.0/14
lan-net 10.0.1.0/24 10.0.2.0/24 10.4.1.0/24
By the end of this step, all your regions will be fully operational with SD-WAN/ADVPN configuration, but they will not be
interconnected in any way.
The goal of the rest of this chapter is to describe how to interconnect them properly, in order for them to become part of a
single SD-WAN solution. See Interconnecting regions on page 98.
Interconnecting regions
Interconnecting regions requires the configuration described in the following topics:

1. Configure overlay and routing. See Configuring overlay and routing on page 98.
2. Configure firewall policies. See Configuring firewall policies on page 109.
Configuring overlay and routing
Interconnecting regions allows for communication between edge devices in different regions.
We will interconnect the regions by using a mix of CLI Templates and ad-hoc CLI Scripts. A full mesh of overlay tunnels
must be built between all the Regional Hubs, over each available underlay transport:

We will establish EBGP session over each tunnel, in order to exchange regional summaries between the Hubs. The
Hubs will then advertise these summaries further to their local Edge devices. The following figure shows an example of
end-to-end advertisement of a particular LAN prefix between the two regions:
As a result, inter-regional connectivity will be provided via the Regional Hubs.

Following is a summary of how to configure overlay and routing:
1. Create a CLI template to summarize and advertise the region’s LAN network. See Creating a CLI Template on page
99.
These network summaries are exchanged via eBGP sessions between the regional hubs.
2. Add the new CLI template to the hub CLI template group, and install the group. See Adding CLI templates to the CLI
Template Group installing on page 100.
3. Create 3 CLI scripts (one for each hub) to configure the inter-hub IPSec tunnels.See Creating CLI scripts on page
101.
The IPSec tunnels secure the data and eBGP sessions between the regional hubs.
4. Run each of the 3 CLI scripts against their respective hubs. See Running CLI scripts on page 109.
Creating a CLI Template
As can be seen, this simple template adds the following configuration:

l It generates an aggregate route for the regional summary (as defined by the lan-summary Meta Field). This route
will be advertised to the remote regions.
l It creates an access list matching this summary, and defines a route-map HUB2HUB_OUT that instructs BGP to
advertise only this summary. This route-map will be used on the inter-regional EBGP peering, as will be shown
further.
To create a CLI Template:
1. Create (or import) a new CLI Template called 04-Hub-Routing-MultiRegion with the following content:
config router bgp
config aggregate-address
edit 1
set prefix $(lan-summary)
next
end
end
config router access-list

edit "REGION_SUMMARY"
config rule
edit 1
set prefix $(lan-summary)
set exact-match enable
next
end
next
end
config router route-map
edit "HUB2HUB_OUT"
config rule
edit 1
set match-ip-address "REGION_SUMMARY"
next
edit 100
set action deny
next
end
next
end
Adding CLI templates to the CLI Template Group installing
To add CLI Templates to a CLI Template Group:
1. Add the new CLI template to the group Hub-Template created earlier.
2. Install the configuration on all the Hubs by using the Quick Install (Device DB) method.

Creating CLI scripts
Now we will use CLI Scripts to create all the necessary IPSEC tunnels between the Hubs. We will add a CLI script per
Hub.
Each script will configure the following:
l Two new loopback interfaces (lo-BGP_INET and lo-BGP_MPLS) will be used for EBGP session termination
between the Hubs. (Note that we use a separate loopback for each type of overlay.)
l IPSEC tunnels to the remote Hubs, over each available underlay transport. In our example, Region 2 Hub will build
two tunnels to each of the Region 1 Hubs (4 tunnels in total). Note that each IPSEC tunnel uses exchange-ip-
addr4 feature to let the remote Hub inject the route to the loopback address (added above). This guarantees that
the loopback addresses are reachable between the Hubs.
l All the newly added IPSEC tunnels are grouped into a new System Zone called hub2hub-overlay.
l EBGP session over each of the above IPSEC tunnels, terminated on one of the loopback interfaces added above.
This session will use HUB2HUB_OUT route-map added earlier, in order to advertise only the regional summary
route to the remote Hubs.
l Policy routes to extend overlay stickiness across the regions. This is to ensure that when a local Edge device
selects a certain type of overlay (for example, overlay over the Internet), the traffic continues on its way to the
remote region via the same type of overlay. Note that the remote Hub will also have its own policy routes in place,
ensuring that the traffic continues on its last hop towards the target Edge device by using the same type of overlay
as well. Hence, this way we achieve an end-to-end overlay stickiness.
To create a CLI scripts:
1. In Device Manager, navigate to Scripts, and create (or import) the CLI scripts.
Following is the content of the CLI scripts for each of the 3 Hubs in our example.
site1-H1 (Region 1, Primary Hub)
edit "lo-BGP_INET"
set vdom "root"
set ip 10.201.0.1 255.255.255.255
set type loopback
next
edit "lo-BGP_MPLS"
set vdom "root"
set ip 10.202.0.1 255.255.255.255
set type loopback
next
end
edit "SITE2-H1_INET"
set interface port1
set ike-version 2
set keylife 28800
set peertype any
set remote-gw 100.64.3.1
set exchange-ip-addr4 10.201.0.1

set exchange-interface-ip enable

set dpd on-idle
next
edit "SITE2-H1_MPLS"
set interface port4
set ike-version 2
set keylife 28800
set peertype any
set dpd on-idle
next
end
set phase1name "SITE2-H1_INET"
next
set phase1name "SITE2-H1_MPLS"
next
end
config system zone
edit hub2hub-overlay
set interface "SITE2-H1_INET" "SITE2-H1_MPLS"
next
end
config router bgp
config neighbor
edit 10.201.0.3
set ebgp-enforce-multihop enable
set interface "lo-BGP_INET"
set update-source "lo-BGP_INET"
set route-map-out "HUB2HUB_OUT"
set connect-timer 1
set remote-as 65002
next
edit 10.202.0.3


set interface "lo-BGP_MPLS"
set update-source "lo-BGP_MPLS"
set connect-timer 1
set remote-as 65002
next
end
end
edit 11
set output-device "SITE2-H1_INET"
set dst 10.0.0.0/8
next
edit 12
set input-device "SITE2-H1_INET"
set dst 10.0.0.0/8
next
edit 13
set output-device "SITE2-H1_MPLS"
set dst 10.0.0.0/8
next
edit 14
set input-device "SITE2-H1_MPLS"
set dst 10.0.0.0/8
next
end
site1-H2 (Region 1, Secondary Hub)

edit "lo-BGP_INET"
set vdom "root"
set ip 10.201.0.2 255.255.255.255
set type loopback
next
edit "lo-BGP_MPLS"
set vdom "root"
set ip 10.202.0.2 255.255.255.255
set type loopback
next
end
set interface port1
set ike-version 2
set keylife 28800
set peertype any


set dpd on-idle
next
set interface port4
set ike-version 2
set keylife 28800
set peertype any
set dpd on-idle
next
end
next
next
end
config system zone
set interface "SITE2-H1_INET" "SITE2-H1_MPLS"
next
end
config router bgp
config neighbor
edit 10.201.0.3
set connect-timer 1
set remote-as 65002

next
edit 10.202.0.3
set connect-timer 1
set remote-as 65002
next
end
end
edit 11
set dst 10.0.0.0/8
next
edit 12
set dst 10.0.0.0/8
next
edit 13
set dst 10.0.0.0/8
next
edit 14
set dst 10.0.0.0/8
next
end
site2-H1 (Region 2, Hub)

edit "lo-BGP_INET"
set vdom "root"
set ip 10.201.0.3 255.255.255.255
set type loopback
next
edit "lo-BGP_MPLS"
set vdom "root"
set ip 10.202.0.3 255.255.255.255
set type loopback
next
end
set interface port1
set ike-version 2

set keylife 28800

set peertype any
set dpd on-idle
next
set interface port4
set ike-version 2
set keylife 28800
set peertype any
set dpd on-idle
next
set interface port1
set ike-version 2
set keylife 28800
set peertype any
set dpd on-idle
next
set interface port4
set ike-version 2
set keylife 28800
set peertype any

set dpd on-idle
next
end
next
next
next
next
end
config system zone
set interface "SITE1-H1_INET" "SITE1-H1_MPLS" "SITE1-H2_INET" "SITE1-H2_MPLS"
next
end
config router bgp
config neighbor
edit 10.201.0.1
set connect-timer 1
set remote-as 65001
next
edit 10.202.0.1
set connect-timer 1

set remote-as 65001

next
edit 10.201.0.2
set connect-timer 1
set remote-as 65001
next
edit 10.202.0.2
set connect-timer 1
set remote-as 65001
next
end
end
edit 11
set dst 10.0.0.0/8
next
edit 12
set dst 10.0.0.0/8
next
edit 13
set dst 10.0.0.0/8
next
edit 14
set dst 10.0.0.0/8
next
edit 15
set dst 10.0.0.0/8
next
edit 16
set dst 10.0.0.0/8
next

edit 17
set dst 10.0.0.0/8
next
edit 18
set dst 10.0.0.0/8
next
end
Running CLI scripts
To run CLI scripts:
1. Run the CLI scripts on respective Hubs (one script per Hub):
Configuring firewall policies
The existing policy package on the Hubs do not account for inter-regional communication. Two rules are expanded and
one new rule is defined.
We must also update the Hub firewall policy.
Following is a summary of how to configure the firewall policy:
1. Create three normalized interfaces for the two BGP loopback interfaces and the hub to hub zone. See Creating
interfaces on page 110.
These loopback interfaces are used for eBGP peering.

2. Edit 2 rules to include the inter-regional zone. See Editing rules on page 110.
This allows communication between the zones.
3. Define a new rule to permit the inter-regional BGP sessions using the normalized bgp loopback interfaces. See
Defining a new rule on page 110.
BGP session traffic must be explicitly permitted since it uses different interfaces than other traffic.
4. Deploy the updated policy package. See Deploying updated policy package on page 111.
Creating interfaces
1. Navigate to Policy & Objects > Object Configurations, and add the new Normalized Interfaces mapped to the newly
created loopback interfaces (lo-BGP_INET, lo-BGP_MPLS), as well as for the zone hub2hub-overlay:
Editing rules
To edit rules:
1. Edit the Edge-Edge and Edge-Hub firewall rules in the Hub policy package, adding the new inter-regional zone to
them, in order to permit inter-regional communication:
Edge- Edge overlay overlay CORP_ CORP_ ALL No Accept

hub2hub - hub2hub - LAN LAN
overlay overlay
Edge-hub lan overlay lan overlay CORP_ CORP_ ALL No Accept

hub2hub - hub2hub - LAN LAN
overlay overlay
Defining a new rule
To define a new rule:
1. Add a new firewall rule, in order to permit inter-regional BGP sessions.

Use the following options for the multi-region firewall rule:

Option Setting
From lo-BGP_INET lo-BGP_MPLS hub2hub-overlay
To lo-BGP_INET lo-BGP_MPLS hub2hub-overlay
Src all
Dst all
Service PING
BGP
NAT No
Action Accept
Deploying updated policy package
To deploy the updated policy package:
1. Reinstall policy on all the Hubs.
Configuring Edge devices with SD-WAN
As can be seen, no configuration change was required on the Edge devices when switching from a single-regional to a
multi-regional deployment.
The Edge devices do not require any configuration related to the remote regions. They only learn the route summaries
from the remote regions, which allows them to send traffic via their local Hubs.
The same is true for SD-WAN configuration. The overlay selection is done locally by the originating Edge device, and the
overlay stickiness policy routes on the Hubs ensure that this selection is preserved end-to-end.

Depending on the requirements and on the nature of the inter-regional communication, more
complex deployments may be possible. These include, but are not limited to:
l Using SD-WAN rules on the Hubs to select the best path towards the remote Hub, rather
than extending the overlay stickiness policy across the regions. This implies that the type
of the overlay might change along the path, if the network conditions justify that.
l Extending ADVPN across the regions to allow direct ADVPN shortcuts between the Edge
devices from different regions. Note that this feature significantly complicates the routing
design, and hence it must be used with care.
These types of deployment are outside the scope of this document.
Make sure that all the inter-regional IPSEC tunnels between the Hubs are up:
On the Edge devices, make sure they learn the summary route of the remote region via BGP:

Finally, the following CLI outputs are shown for reference from one of the Hub devices in our example, demonstrating the
inter-regional connectivity:
Overlay:
site1-H1 # get ipsec tunnel list | grep SITE2
SITE2-H1_INET 100.64.3.1:0 0.0.0.0/0.0.0.0 0.0.0.0/0.0.0.0 up 2615
SITE2-H1_MPLS 172.16.3.5:0 0.0.0.0/0.0.0.0 0.0.0.0/0.0.0.0 up 2619
Routing:
site1-H1 # get router info bgp summary | grep 65002
10.201.0.3 4 65002 169 176 8 0 0 00:12:19 1
10.202.0.3 4 65002 170 171 7 0 0 00:12:21 1
site1-H1 # get router info routing-table bgp | grep SITE2

B 10.4.0.0/14 [20/0] via 10.202.0.3 (recursive via 10.202.0.3, SITE2-
H1_MPLS), 00:12:36 [20/0] via 10.201.0.3 (recursive via 10.201.0.3, SITE2-H1_INET), 00:12:36

Additional topics
This chapter will guide you through several additional topics and SD-WAN features that you can add on top of the
topologies described in the previous chapters.
l Signaling SLA status to hubs on page 114
l SD-WAN routing logic on page 122
Signaling SLA status to hubs
The configuration presented so far applies SD-WAN Rules to both Edge-to-Hub and Edge-to-Edge traffic. But sessions
originated behind the Hub (Hub-to-Edge traffic) were not subject to SD-WAN processing.
Hub-to-Edge traffic may include connections originated by services hosted behind the Hub. (The amount of such traffic is
typically low, because it is more common that Edge sites initiate these connections, which is then Edge-to-Hub traffic.)
But it may also include, for example, connections initiated by the “legacy” sites - those outside of the SD-WAN solution.

Additional topics
Since the Hub has more than one overlay path towards each Edge, it is important to select the best one, considering its
current health status. One option to achieve this goal would be configuring Performance SLA (health checks) on the
Hubs, letting them probe the Edge devices.
We strongly discourage you from using this method, because it violates the Zero-Touch
nature of Hub configuration: the Hub should not have any per-Edge configuration.
This section demonstrates a better approach, where the Edge devices signal the SLA status of their overlays to the Hub
by using BGP. With this approach, the Hub can select the right overlay for Hub-to-Edge traffic, without actively probing
the Edge devices.
We will add the following functionality to the Single Hub deployment on page 24:

Additional topics
l Configuring Edge devices on page 116

l Configuring Hub devices on page 118
l Verifying configuration on page 121
The first part of the solution is to configure the Edge to signal the SLA status of its overlays to the Hub.
Create (or import) a new CLI Template, called 04-Edge-Signal-SLA:
edit "SLA_OK"
config rule
edit 1
set set-community "$(as):99"
next
end
next
end
config router bgp
config neighbor
set route-map-out-preferable "SLA_OK"
next
set route-map-out-preferable "SLA_OK"
next
end
end
This CLI Template defines a new route-map SLA_OK that will attach a custom BGP Community (65001:99 in our
example) to the BGP routes advertised by the Edge over the overlay that meets the SLA. If an overlay does not meet the
SLA, the BGP Community will not be attached. This will allow the Hub to distinguish between healthy and unhealthy
overlays, as will be shown shortly.
In order to achieve this goal, we apply the new route-map as route-map-out-preferable to all the BGP sessions towards
the Hub.
l The route-map-out-preferable will be applied to all the outgoing BGP routes, when the respective BGP
neighbor is marked as healthy (meets the SLA). In this case our custom BGP Community will be attached to the
routes.
l When the respective BGP neighbor is not healthy (out of SLA), the regular route-map-out will be applied instead.
Since we do not configure it, then nothing will be applied, and no BGP Community will be attached to the routes.
Add the new CLI template to the group Edge-Template applied to the Edge devices:

Additional topics
What is still missing is the mechanism that will mark the BGP neighbors as healthy or not healthy, based on the current
SLA status of their respective overlays. We will configure this mechanism in our SD-WAN Template:
l Navigate to SD-WAN > BGP Neighbors, and define two BGP Neighbors BGP_INET and BGP_MPLS that will map
to the neighbor IPs configured on the Edge devices. (Keep their Role as standalone.)
BGP Neighbor Neighbor IP
BGP_INET 10.201.1.1
BGP_MPLS 10.202.1.1
l Edit the SD-WAN Template Edge-Template (assigned to the Edge devices). Add the two BGP Neighbors to it,
linking each one of them to the HUB Performance SLA. Make sure you use the right overlay member for each BGP

Additional topics
Neighbor:
The above configuration can be literally read as follows:

l BGP_INET neighbor is healthy, if H1_INET member is healthy, according to the SLA
target 1 of the HUB Performance SLA

l BGP_MPLS neighbor is healthy, if H1_MPLS member is healthy, according to the SLA
target 1 of the HUB Performance SLA

Recall that when a neighbor is healthy, route-map-out-preferable is applied to the
routes advertised to it. This should now complete the whole picture.
Save the changes and install the configuration on all the Edge devices by using the Quick
Install (Device DB) method.
Configuring Hub devices
The second part of the solution is to configure the Hub, so that it makes use of the information signaled by the Edge.
Create (or import) a new CLI Template, called 04-Hub-Signal-SLA:
config router community-list
edit "SLA_OK"
config rule
edit 1
set action permit
set match "$(as):99"
next
end
next
end
edit "EDGE_IN"
config rule
edit 1

Additional topics
set match-community "SLA_OK"

set set-route-tag 99
next
edit 100
next
end
next
end
config router bgp
config neighbor-group
edit "EDGE"
set route-map-in "EDGE_IN"
next
end
end
This CLI Template defines the same custom BGP Community on the Hub side and uses a new route-map EDGE_IN to
assign a custom route-tag (99 in our example) to the routes with that Community attached. This route-map is applied
as route-map-in to all the routes learned from the Edge devices.
As a result, all the routes learned via healthy overlays will be assigned the route tag 99 (in our example).
Add the new CLI template to the group Hub-Template applied to the Hub:
One common idea is to apply a higher local-preference (or weight) to the routes marked with
SLA_OK community, so that BGP on the Hub prefers them over the routes via unhealthy
overlays. While this approach would work in a basic case, we do not recommend it.
Recall that in the SD-WAN solution, the duty to steer the traffic must be delegated to SD-WAN!
We would like BGP to keep all available routes and let SD-WAN select the preferred path for
each case. For example, certain non-business-critical traffic might be still steered via an
unhealthy (but cheaper) overlay. Whether it is Edge-to-Hub or Hub-to-Edge traffic, it requires a
valid route on the Hub via the unhealthy overlay. Giving this route lower local-preference (or
weight) would remove it from the routing table, thus violating this requirement.

Additional topics
We will now configure SD-WAN Rules on the Hub that will prefer the healthy overlays, based on the route tags:
l Edit the SD-WAN Template Hub-Template (assigned to the Hub).
l Create an SD-WAN Rule called INET_OK, matching the Route Tag as Destination and explicitly specifying the
EDGE_INET member for it (using Manual strategy):
l Similarly, create another SD-WAN Rule called MPLS_OK, matching the same Route Tag as Destination and
explicitly specifying the EDGE_MPLS member for it (using Manual strategy):
Save the changes and install the configuration on the Hub by using the Quick Install (Device DB) method.

Additional topics
The following CLI command on the Hub will confirm that the routes from Edge devices are marked with the SLA_OK
Community and therefore get the right route tag:
site1-H1 # get router info bgp network 10.0.1.0/24
VRF 0 BGP routing table entry for 10.0.1.0/24
Paths: (2 available, best #2, table Default-IP-Routing-Table)
Advertised to peer-groups:
EDGE
Original VRF 0
Local, (Received from a RR-client)
10.202.1.2 from 10.202.1.2 (10.0.1.1)
Origin IGP metric 0, route tag 99, localpref 100, valid, internal, best
Community: 65001:99
Advertised Path ID: 2
Last update: Thu Sep 2 03:10:16 2021
Original VRF 0
10.201.1.2 from 10.201.1.2 (10.0.1.1)
Community: 65001:99
In this situation, the first configured SD-WAN Rule (INET_OK) will match this traffic, and therefore it will be forwarded via
the INET overlay, as expected. This can be confirmed by the following CLI output:
site1-H1 # diagnose sys sdwan service 1
Service(1): Address Mode(IPV4) flags=0x200

Gen(3), TOS(0x0/0x0), Protocol(0: 1->65535), Mode(manual)
Members(1):
1: Seq_num(2 EDGE_INET), alive, selected
Src address(1):
10.0.0.0-10.255.255.255
Route tag address(2):

10.0.1.0-10.0.1.255
10.0.2.0-10.0.2.255
However, if H1_INET overlay on “site1-1” device goes out of SLA, the corresponding route will no longer receive the
route tag:
site1-H1 # get router info bgp network 10.0.1.0/24
VRF 0 BGP routing table entry for 10.0.1.0/24
Paths: (2 available, best #2, table Default-IP-Routing-Table)
Advertised to peer-groups:
EDGE
Original VRF 0
10.202.1.2 from 10.202.1.2 (10.0.1.1)
Community: 65001:99

Additional topics
Original VRF 0
10.201.1.2 from 10.201.1.2 (10.0.1.1)
Origin IGP metric 0, localpref 100, valid, internal, best (no route tag!)
As a result, this prefix will no longer appear in the first SD-WAN Rule:

Members(1):
1: Seq_num(2 EDGE_INET), alive, selected
Src address(1):
10.0.0.0-10.255.255.255
Route tag address(1): (no 10.0.1.0/24 prefix!)

10.0.2.0-10.0.2.255
Hence, the traffic will skip this rule and match the next one (“MPLS_OK”). The traffic will be forwarded via EDGE_MPLS
overlay, as expected:

Members(1):
1: Seq_num(3 EDGE_MPLS), alive, selected
Src address(1):
10.0.0.0-10.255.255.255
Route tag address(2):

10.0.1.0-10.0.1.255
10.0.2.0-10.0.2.255
SD-WAN routing logic
In this guide, we have recommended to configure SD-WAN as a default route. Let us see why it helps making the
configuration generic and what alternatives you may want to consider.
The SD-WAN / SD-Branch Reference Architecture for MSSPs describes the interaction between SD-WAN and the
traditional routing subsystem. Let us recap the two main rules that apply by default:
1. SD-WAN Rules are matched only if the best route to the destination points to SD-WAN.
2. SD-WAN Member is selected only if it has a valid route to the destination (not necessarily the best route).
Both these rules can be disabled by using advanced options in SD-WAN rules:
l Rule #1 is controlled by the advanced option default (corresponding to CLI set default enable)
l Rule #2 is controlled by the advanced option gateway (corresponding to CLI set gateway enable)
According to rule #2, by default, SD-WAN rules select a member only if there is a valid route to destination via that
member. For Edge-to-Hub and Edge-to-Edge traffic, this valid route will normally be learned via BGP. However, for
Edge-to-Internet traffic, no specific route is learned. Hence, for example, in order for the RIA rule to work as desired in

Additional topics
our examples, it is required to have a default gateway via T_MPLS overlay. Otherwise the traffic destined to the Internet
would never be backhauled via T_MPLS.
Configuring SD-WAN to act as a default route eliminates the need to adjust the routing configuration when your SD-WAN
rules change. It ensures that there always be a valid route to any destination via any SD-WAN member that is selected
by the SD-WAN rules. Thus, SD-WAN rules become fully responsible for traffic steering, in accordance with the Five-
Pillar Design Approach.
It is worth noting a few alternatives to this approach:
l Using the default route via underlay on page 123
l Disabling route check in SD-WAN rules on page 124
l Excluding Traffic from SD-WAN on page 125
Using the default route via underlay
If RIA functionality is not required, it is enough to only keep the default route via the underlay. (For example, your
standard default route received via DHCP from an ISP router.)
l Edge-to-Internet traffic will be able to breakout locally, using this default route.
l Edge-to-Hub and Edge-to-Edge traffic will use the specific routes learned via BGP.
No other routing configuration is needed.
If any other destination exists that must be reached via the overlays (either Internet destination
or a legacy site outside the SD-WAN solution), you must ensure that a valid route to that
destination via the required overlay exists. This route can be either learned dynamically or
configured statically.

Additional topics
Disabling route check in SD-WAN rules
Another alternative to using SD-WAN as a default route globally is to disable route check on per-rule basis.
For each SD-WAN rule where a valid route to the destination is not expected to exist (such as the RIA rules), you can
enable the two advanced options gateway and default, as mentioned in the note in Using the default route via underlay
on page 123.
This instructs the SD-WAN rule to bypass any route check, and forward the traffic unconditionally via the member
selected by the configured strategy. Hence, if T_MPLS is selected in our RIA example, the Internet traffic will be
backhauled via the Hub even if there is no default route learned via T_MPLS.

Additional topics
Excluding Traffic from SD-WAN
Note that even if you configure SD-WAN to act as a default route, as recommended throughout this guide, you can still
exclude certain traffic from SD-WAN processing. The rule #1 helps you to achieve that.
By default, before selecting a member, SD-WAN rule will check whether the best route to the destination points to any
SD-WAN member at all. If it doesn’t, this traffic will considered as out of SD-WAN scope, and hence it will be handled by
the traditional routing.
A good use case is out-of-band management (OOB). As shown on the below diagram, SD-WAN acts as a default route,
but there are more specific management prefixes learned via the OOB network.
This guarantees that all the traffic destined to those prefixes bypasses SD-WAN rule processing and is forwarded to the
OOB network.

Appendix A - Products used
The following product models and firmware were used in this guide:
Product Model Firmware
FortiOS All models supported by 6.4.6

FortiManager
FortiManager All models 6.4.6

Copyright© 2021 Fortinet, Inc. All rights reserved. Fortinet®, FortiGate®, FortiCare® and FortiGuard®, and certain other marks are registered trademarks of Fortinet, Inc., in the
U.S. and other jurisdictions, and other Fortinet names herein may also be registered and/or common law trademarks of Fortinet. All other product or company names may be
trademarks of their respective owners. Performance and other metrics contained herein were attained in internal lab tests under ideal conditions, and actual performance and
other results may vary. Network variables, different network environments and other conditions may affect performance results. Nothing herein represents any binding
commitment by Fortinet, and Fortinet disclaims all warranties, whether express or implied, except to the extent Fortinet enters a binding written contract, signed by Fortinet’s
General Counsel, with a purchaser that expressly warrants that the identified product will perform according to certain expressly-identified performance metrics and, in such
event, only the specific performance metrics expressly identified in such binding written contract shall be binding on Fortinet. For absolute clarity, any such warranty will be
limited to performance in the same ideal conditions as in Fortinet’s internal lab tests. In no event does Fortinet make any commitment related to future deliverables, features or
development, and circumstances may change such that any forward-looking statements herein are not accurate. Fortinet disclaims in full any covenants, representations, and
guarantees pursuant hereto, whether express or implied. Fortinet reserves the right to change, modify, transfer, or otherwise revise this publication without notice, and the most
current version of the publication shall be applicable.

Secure SD-WAN Deployment Guide For MSSPs

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Secure SD-WAN Deployment Guide For MSSPs

Uploaded by

Copyright:

Available Formats

Secure SD-WAN

Deployment Guide for MSSPs

FORTINET VIDEO GUIDE

CUSTOMER SERVICE & SUPPORT

FORTINET TRAINING & CERTIFICATION PROGRAM

END USER LICENSE AGREEMENT

Secure SD-WAN 6.4 Deployment Guide for MSSPs 3

Secure SD-WAN 6.4 Deployment Guide for MSSPs 4

Date Change Description

2021-11-04 Initial release.

Secure SD-WAN 6.4 Deployment Guide for MSSPs 5

About this guide

Secure SD-WAN 6.4 Deployment Guide for MSSPs 6

You must complete each section before moving on to subsequent sections.

Secure SD-WAN 6.4 Deployment Guide for MSSPs 7

Use cases and topologies

Single Hub topology

Secure SD-WAN 6.4 Deployment Guide for MSSPs 8

Dual Hub topology

Secure SD-WAN 6.4 Deployment Guide for MSSPs 9

This section describes how to extend the topology to multiple regions:

Secure SD-WAN 6.4 Deployment Guide for MSSPs 10

Design concepts and considerations

Secure SD-WAN 6.4 Deployment Guide for MSSPs 11

Secure SD-WAN 6.4 Deployment Guide for MSSPs 12

Secure SD-WAN 6.4 Deployment Guide for MSSPs 13

l Option 2: Deployed on MSSP premises, dedicated to end-customer, no multitenancy (usually virtual)

l Option 3: Deployed on end-customer premises (Enterprise design), no multitenancy

instance, when it is dedicated to end-customer).

designed for the end-customers.

FMG/FAZ by using REST API, thanks to the comprehensive automation support.

Secure SD-WAN 6.4 Deployment Guide for MSSPs 14

The deployment overview includes the following sections:

l Enable central SD-WAN management.

Secure SD-WAN 6.4 Deployment Guide for MSSPs 15

Configuring the root ADOM

Secure SD-WAN 6.4 Deployment Guide for MSSPs 16

To configure the root ADOM:

1. In FortiManager, go to System Settings > All ADOMs.

3. Beside Central Management, select SD-WAN, and click OK.

Enabling and creating ADOMs

1. In FortiManager, enable ADOMs:

Secure SD-WAN 6.4 Deployment Guide for MSSPs 17

3. Click the Create New button.

4. In the Name box, type a name for the ADOM.

Secure SD-WAN 6.4 Deployment Guide for MSSPs 18

5. In the Type box, select FortiGate, and click 6.4.

Creating device groups

In Device Manager, create the following device groups:

To create device groups:

1. In FortiManager, go to Device Manager > Device & Groups.

Creating a default system template

To create a system template:

1. In FortiManager, go to Device Manager > Provisioning Templates > System Templates.

Secure SD-WAN 6.4 Deployment Guide for MSSPs 19

4. Double-click the template to open it for editing.

Creating certificate templates

This section describes how to create the following certificate templates:

To create a certificate template:

Secure SD-WAN 6.4 Deployment Guide for MSSPs 20

3. In the Certificate Name box, type Edge.

Creating normalized interfaces

Create the following normalized interfaces: