Fortigate Infrastructure: High Availability (Ha)

FortiGate Infrastructure
High Availability (HA)
FortiOS 7.0
© Copyright Fortinet Inc. All rights reserved. LastLast
Modified:
Modified:
Friday,
Friday,
JulyJuly
29, 29,
20222022
Lesson Overview
HA Operation Modes
HA Cluster Synchronization
HA Failover and Workload
Monitoring and Troubleshooting

© Fortinet Inc. All Rights Reserved. 2
HA Operation Modes
Objectives
• Identify the different operation modes for HA
• Understand the primary FortiGate election in an HA cluster
3
What Is FortiGate HA?
Two or more FortiGate devices

operate as an HA cluster
Switch
FortiGate Devices
Switch

Active-Passive HA
Configuration of the primary
All secondaries are on
FortiGate is synchronized with
standby
one or more secondaries
Secondaries
Only the primary

Primary processes traffic
If primary fails, a
secondary takes over

Active-Active HA
All FortiGate devices process

traffic
The primary distributes

the sessions to the cluster
members
If primary fails, a secondary

takes the session distribution job

FortiGate Clustering Protocol (FGCP)
• A cluster uses FortiGate clustering protocol (FGCP) to:
• Discover other FortiGate devices that belong to the same HA group
• Elect the primary
• Synchronize configuration and other data
• Detect when a FortiGate fails
• FGCP runs only over the heartbeat links

• Uses TCP port 703 with different Ethernet type values
• 0x8890 – NAT mode
• 0x8891 – transparent mode
• Uses TCP port 23 with Ethernet type 0x8893 for configuration synchronization
• If the primary FortiGate is rebooted or shut down, it becomes the secondary FortiGate
and waits for the traffic to failover to the new primary, before it reboots or shuts down

HA Requirements
• Two to four identical FortiGate devices
• Same licenses on all cluster members
• One link (preferably two or more) between FortiGate devices for heartbeat
• Same interfaces on each FortiGate connected to the same broadcast domain
• DHCP and PPPoE interfaces are supported
Port 1 Port 2
Heartbeat 1 Heartbeat 2
Port 1 Port 2

Primary FortiGate Election: Override Disabled
Begin Negotiation
• Override disabled (default)
• Force a failover Greater Connected Less
• diagnose sys ha reset-uptime Monitored
ports
• Check the HA uptime differences
Difference measured in 1/10 second Greater Less
HA Uptime
# diagnose sys ha dump-by vcluster Greater Less

... Priority
FGVMxxxx92:...uptime/reset_cnt=7814/0
FGVMxxxx36:...uptime/reset_cnt=0/1
Greater Serial Less

0 is for the device Number
Number of times HA
with lower HA uptime
uptime has been reset
for this device
Primary FortiGate Secondary FortiGate

Primary FortiGate Election: Override Enabled
Begin Negotiation
• Override enabled
config system ha
set override enable Greater Connected Less
Monitored
end ports
• Force a failover
• Change the HA priority Greater Less
HA Uptime
Priority
Greater Less
HAPriority
Uptime
Greater Serial Less

Number
Primary FortiGate Secondary FortiGate

Knowledge Check
1. To form an HA cluster, all FortiGate devices that will be included in the cluster must
have which of the following?
A. The same FortiGate hostname
B. The same firmware
2. What is the default criteria (override disabled) for selecting the HA primary device in an
HA cluster?
A. Connected monitored ports > HA uptime > priority > serial number
B. Priority > HA uptime > connected monitored ports > serial number

Lesson Progress
HA Operation Modes

Objectives
• Identify the primary and secondary device tasks in an HA
cluster
• Identify what is synchronized between HA cluster members
• Configure session synchronization for seamless failover
13
Primary FortiGate Tasks
• Exchanges heartbeat hello packets with all the secondary devices
• Synchronizes its routing table, DHCP information, and part of its configuration to all the
secondary devices
• Can synchronize the information of some of the traffic sessions for seamless failover
• In active-active mode only:
• Distributes specific traffic among all the devices in the cluster

Secondary FortiGate Tasks
• Monitors the primary for signs of failure using hello or port monitoring
• If a problem is detected with the primary, the secondary devices elect a new primary
• In active-active mode only:

• Processes traffic distributed by the primary

Heartbeat Interface IP Addresses
• The cluster assigns virtual IP addresses to heartbeat interfaces based on the serial
number of each FortiGate device:
• 169.254.0.1: for the highest serial number
• 169.254.0.2: for the second highest serial number
• 169.254.0.3: for the third highest serial number (and so on)
• FortiGate devices keep their heartbeat virtual IP addresses regardless of any change in
their role (primary or secondary)
• The IP address assignment changes only when a FortiGate leaves or joins cluster
• Cluster uses these virtual IP addresses to:
• Distinguish the cluster members
• Update configuration changes to the cluster members

Heartbeat Ports and Monitored Ports
• Heartbeat ports contain sensitive cluster configuration information
• Must have one heartbeat interface, but using two for redundancy is recommended
• Cannot use FortiGate switch port for heartbeat port
• Monitored ports are usually networks (interfaces) processing high priority traffic
• Avoid configuring interface monitoring for all interfaces
• Do not monitor dedicated heartbeat interfaces
• Can monitor VLAN interfaces
• Wait until a cluster is up and running and all interfaces are connected before enabling interface
monitoring

HA Complete Configuration Synchronization
Complete Synchronization
1. New secondary is
2. Primary compares its checksum of configuration added to cluster
with the new secondary checksum. If it is
different, it sends its configuration
New secondary
Configuration
Primary

HA Incremental Configuration Synchronization
Incremental Synchronization
2. Change is synchronized
to secondary
1. Primary configuration is
changed
Secondary
Configuration
Primary
Configuration

HA Configuration Synchronization
• Incremental synchronizations also include:
• Dynamic data such as DHCP leases, routing table updates, IPsec SAs, session information, and so on
• Periodically, HA checks for synchronization
• If CRC checksum values match, cluster is in sync
• If checksums don’t match after five attempts, secondary downloads the whole configuration from the
primary

What Is Not Synchronized?
• These configuration settings are not synchronized between cluster members:
• HA management interface settings
• HA default route for the reserved management interface
• In-band HA management interface
• HA override
• HA device priority
• HA virtual cluster priority
• FortiGate hostname
• Ping server HA priorities
• HA priority (ha-priority) setting for a ping server or dead gateway detection configuration
• Licenses
• FortiGuard, FortiCloud activation, and FortiClient licensing
• Cache
• FortiGuard Web Filtering and email filter, web cache, and so on
• The primary FortiGate synchronizes all other configuration settings and other
configuration details related to HA settings

Session Synchronization
• You can enable session table synchronization
for most TCP and IPsec VPN sessions
config system ha
• You can enable synchronization only for sessions not
set session-pickup enable
being handled by proxy-based security profiles end
• You can enable synchronization for UDP and config system ha

set session-pickup enable
ICMP sessions set session-pickup-connectionless enable
end
• You can enable synchronization for multicast config system ha

sessions set multicast-ttl <5 – 3600 sec>
end
• You cannot enable synchronization for SSL

VPN sessions

Knowledge Check
1. Which information is synchronized between two FortiGate devices that belong to the
same HA cluster?
A. Firewall policies and objects
B. FortiGate hostname
2. Which one of the following session types can be synchronized in an HA cluster?

A. SSL VPN sessions
B. IPsec VPN sessions

Lesson Progress
HA Operation Modes

Objectives
• Identify the HA failover types
• Interpret how an HA cluster in active-active mode distributes
traffic
• Implement virtual clustering per virtual domain (VDOM) in an
HA cluster
25
Failover Protection Types
• Device failover
• If the primary stops sending heartbeat packets, another FortiGate automatically takes its place
• Link failover
• The cluster can monitor some interfaces to determine if they are operating and connected
• If a monitored interface on the primary fails, the cluster elects a new primary
• Session failover
• When session pickup is enabled, the newly elected primary resumes active session, avoiding the need
to restart active session
• Memory utilization failover
• When configured, an HA failover can be triggered when memory utilization exceeds the threshold for a
specific amount of time
• Event logs, SNMP traps, and alert email record failover events

Virtual MAC Addresses and Failover
• On the primary, each interface is assigned a virtual MAC address
• HA heartbeat interfaces are not assigned a virtual MAC address
• Upon failover, the newly elected primary adopts the same virtual MAC addresses as the
former primary
HA heartbeat interfaces
Primary
Former primary
Virtual MAC addresses
After failover, gratuitous ARP informs the

network that the virtual MAC address is now
reachable through a different FortiGate
New primary
Secondary

Failure of a Secondary FortiGate
• Active-passive HA cluster
• The primary updates the list of available secondary FortiGate devices
• Active-active HA cluster
• The primary updates the list of available secondary FortiGate devices and redistributes sessions to
prevent failed secondary devices

Workload
• Active-passive HA cluster
• The primary receives and processes all traffic
• The secondary waits passively
• Active-active HA cluster
• The primary receives all traffic and redirects some traffic to secondary devices

Active-Active Traffic Flow (Proxy Inspection)
primary
primary-virtual MAC-port1
port1
port2
primary-physical MAC-port1
1 - SYN
2 - SYN
secondary-physical MAC-port1
Client secondary
3a - SYN
3b – SYN/ACK
port1
port2 Server
1. srcMAC X, dstMAC primary-virtual MAC-port1, TCP SYN dport 80

2. srcMAC primary-physical MAC-port1, dstMAC secondary-physical MAC-port1, TCP SYN dport 80
3a. srcMAC secondary-physical MAC-port2, dstMAC Y, TCP SYN dport 80
3b. srcMAC secondary-physical MAC-port1, dstMAC X, TCP SYN ACK sport 80

Active-Active Traffic Flow (Proxy Inspection) (Contd)
primary
primary-virtual MAC-port1
port1
port2
4 - ACK primary-physical MAC-port1
5 - ACK
Client secondary
port1
port2 Server
4. srcMAC X, dstMAC primary-virtual MAC-port1, TCP ACK dport 80

5. srcMAC primary-physical MAC-port1, dstMAC secondary-physical MAC-port1, TCP ACK dport 80

Active-Active Traffic Flow (Proxy Inspection - Contd)
primary
port1
port2 primary-virtual MAC-port2
primary-physical MAC-port2 6 - SYN/ACK
7 - SYN/ACK
secondary
port1
port2 8 - ACK Server
6. srcMAC Y, dstMAC primary-virtual MAC-port2, TCP SYN ACK sport 80

7. srcMAC primary-physical MAC-port2, dstMAC secondary-physical MAC-port2, TCP SYN ACK sport
80
8. srcMAC secondary-physical MAC-port2, dstMAC Y, TCP ACK dport 80
Active-Active Traffic Flow (No Proxy Inspection)
primary
primary-virtual MAC-port1 primary-virtual MAC-port2
port1 port2
4 - SYN/ACK
1 - SYN
primary-physical MAC port1/port2
2 - SYN 5 – SYN/ACK
8 - ACK
7 - ACK secondary-physical MAC port1/port2
Client
3 - SYN
Server
6 – SYN/ACK
port1 port2 9 - ACK
secondary-physical MAC-port1 secondary-physical MAC-port2
secondary

Virtual Clustering
• Virtual clusters are an extension of FGCP for FortiGate with multiple VDOMs
• HA cluster must consist of only two FortiGate devices
• Allows FortiGate to be the primary for some VDOMs and the secondary for the other
VDOMs
Active-Passive HA
Domain A Domain B Domain C Domain A Domain B Domain C
Primary Primary Secondary Secondary Secondary Primary

Full Mesh HA
• Reduces the number of single points of failure
• Uses aggregate and redundant interfaces for robust connections between all network
components
FortiGate
HB 2
HB 1
FortiGate

Knowledge Check
1. An HA failover occurs when the link status of a monitored interface on the _____ goes
down.
A. Primary FortiGate
B. Secondary FortiGate
2. You can configure virtual clustering between only ____ FortiGate devices with multiple
VDOMs in an active-passive HA cluster.
A. Two
B. Four

Lesson Progress
HA Operation Modes

Objectives
• Verify the normal operation of an HA cluster
• Configure an HA management interface
• Upgrade an HA cluster firmware
38
Checking the Status of the HA Using the GUI
• Add the HA status widget
• Click System > HA

Checking the Status of the HA Using the CLI
# diagnose sys ha status
HA information
.......
......
nvcluster=1, ses_pickup=1, delay=1, load_balance=0, schedule=3, ldb_udp=0, upgrade_mode=0.
Primary and secondary
HA information
[Debug_Zone HA information]
HA group member information: is_manage_primary=1.
FGVM010000112065: Primary, serialno_prio=0, usr_priority=200, hostname=Local-FortiGate
FGVM010000065036: Secondary, serialno_prio=1, usr_priority=100, hostname=Remote-FortiGate
[Kernel HA information]
vcluster 1, state=work, primary_ip=169.254.0.1, primary_id=0:
FGVM010000112065: Primary, ha_prio/o_ha_prio=0/0
FGVM010000065036: Secondary, ha_prio/o_ha_prio=1/1 Heartbeat interface IP 169.254.0.1
assigned to the highest serial number

Checking the Configuration Synchronization
Cluster checksum example
• Run the following command on the cluster
member(s): # diagnose sys ha checksum cluster
========= FGVM010000112065 =================

# diagnose sys ha checksum
is_manage_primary()=1, is_root_primary()=1
cluster Show HA cluster checksum debugzone
global: 7b 05 62 17 8f cd 76 29 57 da 32 8e
show Show HA checksum of logged root: 97 91 80 67 9d 97 e3 a1 dd 0d ca
in FortiGate all: e1 ad dd fb ff f6 e5 55 2c ed 3b
recalculate Re-calculate HA checksum checksum

global: 7b 05 62 17 8f cd 76 29 57 da 32 8e
root: 97 91 80 67 9d 97 e3 a1 dd 0d ca
all: e1 ad dd fb ff f6 e5 55 2c ed 3b
• All peers must have the same sequences of
checksum numbers ========= FGVM010000065036 ================
is_manage_primary()=0, is_root_primary()=0
debugzone
global: 7b 05 62 17 8f cd 76 29 57 da 32 8e
root: 97 91 80 67 9d 97 e3 a1 dd 0d ca
checksum
global: 7b 05 62 17 8f cd 76 29 57 da 32 8e
root: 97 91 80 67 9d 97 e3 a1 dd 0d ca

Switching to the CLI of a Secondary FortiGate
• Using the CLI of the primary FortiGate, you can connect to any secondary CLI:
# execute ha manage <cluster_id> <Admin_Username>
• To list index numbers for each FortiGate device, use a question mark:
# execute ha manage ?
<id> please input peer box index.
<1> Subsidary unit FGVM0100000xxxxx

Force HA Failover for Troubleshooting
• You can force HA failover on a primary device:
# execute ha failover set <cluster_id>
• Device stays in failover state regardless of condition
• Forced failover on primary device:
# execute ha failover set 1
Caution: This command will trigger an HA failover. Should be used for
It is intended for testing purposes.
testing, demo, or
troubleshooting
Do you want to continue? (y/n)y
purposes only!
• To view failover status Do not use in live
# execute ha failover status environments.
failover status: set

Force HA Failover for Troubleshooting (Contd)
• To view the system status of a device in forced HA failover:
# get system ha status
Forced failover was used
HA Health Status: OK
to select primary
………
Primary selected using:
<2020/04/19 10:16:54> FGVM010000064692 is selected as the primary because it has EXE_FAIL_OVER
flag set.
<2020/04/19 10:07:29> FGVM010000065036 is selected as the primary because it has the largest value
of override priority.
• To stop the failover status:

# execute ha failover unset 1
• To view the system status of a device after forced HA failover is disabled:

# get system ha status Primary is selected based on
……… device with highest priority
Primary selected using:
<2020/04/19 10:38:28> FGVM010000065036 is selected as the primary because it has the
largest value of override priority.
<2020/04/19 10:16:54> FGVM010000064692 is selected as the primary because it has
EXE_FAIL_OVER flag set.

Reserved HA Management Interface
• Available in both NAT mode and transparent mode
• Can connect directly and separately to each FortiGate—CLI and GUI
• Can configure up to four dedicated HA management interfaces
• Can configure a different IP address for this interface for each FortiGate
• Configuration changes related to HA management interface are not synchronized with the other
FortiGate devices in an HA cluster
• In-band HA management interface is an alternative to the reserved HA management

interface feature
• Does not require reserving an interface just for management access
• Does not synchronize HA management IP address settings among cluster members
• Configured from the CLI
config system interface

Can use execute ha manage
edit <port name> command to connect to individual
set management-ip <IP address and subnet mask> devices in cluster to configure in-
end band management IP address

Firmware Updates
Current Primary
• To upgrade an HA cluster, you only need to
upload the new firmware to the primary: Local # Wait for HA to be primary of all
clusters.. 1
• Uninterruptable upgrade is enabled by default Send image to HA secondary.
• In active-active mode, traffic load balancing is
Wait for secondary to
temporarily turned off while all devices are upgrading restart................
their firmware
Wait for first secondary to become new
primary.
1.The cluster upgrades the firmware on all the Firmware upgrade in progress....
secondaries Done.
3
2.A new primary is elected 2 Current Secondary(s)
3.The cluster upgrades the firmware in Remote # Get image from ha primary OK.
Check image OK.
the ..former primary Please wait for system to restart.
Firmware upgrade in progress....

Done.
The system is going down NOW !!
System is starting...

Knowledge Check
1. The heartbeat interface IP address 169.254.0.1 is assigned to which FortiGate in an
HA cluster?
A. The FortiGate with the highest serial number
B. The FortiGate with the highest priority
2. Which statement about the firmware upgrade process on an HA cluster is true?

A. You need to upload the new firmware only to the primary FortiGate to upgrade an HA cluster.
B. The cluster members are not rebooted.

Lesson Progress
HA Operation Modes

Review
 Identify the different operation modes for HA
 Understand the primary FortiGate election in an HA cluster
 Identify primary and secondary device tasks in an HA cluster
 Identify what is synchronized between HA cluster members
 Configure session synchronization for seamless failover
 Identify the HA failover types
 Interpret how an HA cluster in active-active mode distributes traffic
 Implement virtual clustering per VDOM in an HA cluster
 Verify the normal operation of an HA cluster
 Configure the HA management interface
 Upgrade an HA cluster firmware

Fortigate Infrastructure: High Availability (Ha)

Uploaded by

Copyright:

Available Formats

You might also like

Fortigate Infrastructure: High Availability (Ha)

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Fortigate Infrastructure: High Availability (Ha)

Uploaded by

Copyright:

Available Formats

FortiGate Infrastructure

High Availability (HA)

HA Failover and Workload

Monitoring and Troubleshooting

Two or more FortiGate devices

© Fortinet Inc. All Rights Reserved. 4

Only the primary

© Fortinet Inc. All Rights Reserved. 5

All FortiGate devices process

The primary distributes

If primary fails, a secondary

© Fortinet Inc. All Rights Reserved. 6

• FGCP runs only over the heartbeat links

© Fortinet Inc. All Rights Reserved. 7

© Fortinet Inc. All Rights Reserved. 8

# diagnose sys ha dump-by vcluster Greater Less

Greater Serial Less

Primary FortiGate Secondary FortiGate

© Fortinet Inc. All Rights Reserved. 9

Greater Serial Less

Primary FortiGate Secondary FortiGate

© Fortinet Inc. All Rights Reserved. 10

© Fortinet Inc. All Rights Reserved. 11

HA Failover and Workload

Monitoring and Troubleshooting

© Fortinet Inc. All Rights Reserved. 12

© Fortinet Inc. All Rights Reserved. 14

• In active-active mode only:

© Fortinet Inc. All Rights Reserved. 15

© Fortinet Inc. All Rights Reserved. 16

© Fortinet Inc. All Rights Reserved. 17

© Fortinet Inc. All Rights Reserved. 18

© Fortinet Inc. All Rights Reserved. 19

© Fortinet Inc. All Rights Reserved. 20

© Fortinet Inc. All Rights Reserved. 21

• You can enable synchronization for UDP and config system ha

• You can enable synchronization for multicast config system ha

• You cannot enable synchronization for SSL

© Fortinet Inc. All Rights Reserved. 22

2. Which one of the following session types can be synchronized in an HA cluster?

© Fortinet Inc. All Rights Reserved. 23

HA Failover and Workload

Monitoring and Troubleshooting

© Fortinet Inc. All Rights Reserved. 24

© Fortinet Inc. All Rights Reserved. 26

After failover, gratuitous ARP informs the

© Fortinet Inc. All Rights Reserved. 27

© Fortinet Inc. All Rights Reserved. 28

© Fortinet Inc. All Rights Reserved. 29

1. srcMAC X, dstMAC primary-virtual MAC-port1, TCP SYN dport 80

© Fortinet Inc. All Rights Reserved. 30

4 - ACK primary-physical MAC-port1

4. srcMAC X, dstMAC primary-virtual MAC-port1, TCP ACK dport 80

© Fortinet Inc. All Rights Reserved. 31

primary-physical MAC-port2 6 - SYN/ACK

6. srcMAC Y, dstMAC primary-virtual MAC-port2, TCP SYN ACK sport 80

© Fortinet Inc. All Rights Reserved. 33

Domain A Domain B Domain C Domain A Domain B Domain C