KUMON Fortigate firewall troubleshooting guide

Front View

Rear View

Common Issues on Fortigate firewall as per IPmon:-

1. FortiGate CPU Utilization

2. FortiGate Memory Usage
3. Interface Errors
4. PING
5. SNMP Health Check
6. SNMP Uptime
7. Host down alert
Troubleshooting steps/guide for: -

1. FortiGate CPU Utilization

For CPU utilization issues, we use “get system performance status” command which will give
overall information of CPU, Memory and Uptime.

You can use the “diagnose sys top” command from the FortiOS CLI to list the processes
running on your FortiGate unit. The command also displays information about each process.
Example output:

CLI# diagnose sys top

Run Time: 13 days, 13 hours and 58 minutes

0U, 0S, 98I; 123T, 25F, 32KF
newcli    903       R       0.5       5.5
sshd      901       S       0.5       4.0

Where the codes displayed on the second output line mean the following:

 U is % of user space applications using CPU. In the example, 0U means 0% of the

user space applications are using CPU.
 S is % of system processes (or kernel processes) using CPU. In the example, 0S
means 0% of the system processes are using the CPU.
 I is % of idle CPU. In the example, 98I means the CPU is 98% idle.
 T is the total FortiOS system memory in Mb. In the example, 123T means there are
123 Mb of system memory.
 F is free memory in Mb. In the example, 25F means there is 25 Mb of free memory.
 KF is the total shared memory pages used. In the example, 32KF means the system
is using 32 shared memory pages.

Each additional line of the command output displays information for each of the processes
running on the FortiGate unit. For example, the third line of the output is:
 newcli    903       R       0.5       5.5

Where:

 newcli is the process name. Other process names can include ipsengine, sshd,
cmdbsrv, httpsd, scanunitd, and miglogd.
 903 is the process ID. The process ID can be any number.
 R is the state that the process is running in. The process state can be:
o R running.
o S sleep.
o Z zombie.
o D disk sleep.
 0.5 is the amount of CPU that the process is using. CPU usage can range from 0.0 for
a process that is sleeping to higher values for a process that is taking a lot of CPU
time.
 5.5 is the amount of memory that the process is using. Memory usage can range
from 0.1 to 5.5 and higher.

Interactive diagnose sys top commands

You can enter the following single-key commands when diagnose sys top is running.

 Press q to quit.
 Press c to sort the processes by the amount of CPU that the processes are using.
 Press m to sort the processes by the amount of memory that the processes are
using.

Stopping running processes

Please be cautious at below steps as below information serve as information to GSD team. Once find
out which process hogging more CPU, forward the information to on-site support team or NTTS
Engineer for further assistance.

You can use the following command to stop running processes:

diagnose sys kill <signal> <process id>

Where:

 <signal> can be any number but 11 is preferred because this signal sends output to the
crashlog which can be used by Fortinet Support to troubleshoot problems.
 <process id> is the process ID listed by the diagnose sys top command.

For example, to stop the process with process ID 903, enter the following command:

diagnose sys kill 11 903

 2. Fortigate Memory usage

FortiOS has a finite set of hardware resources such as memory and all the running processes
share that memory. Depending on their workload, each process will use more or less as
needed, usually more in high traffic situations. If some processes use all the available
memory, other processes will have no memory available and not be able to function.

When high memory usage happens, you may experience services that appear to freeze up
and connections are lost or new connections are refused.

If you are seeing high memory usage in the System Resources widget, it could mean that the
unit is dealing with high traffic volume, which may be causing the problem, or it could be
when the unit is dealing with connection pool limits affecting a single proxy. If the unit is
receiving large volumes of traffic on a specific proxy, it is possible that the unit will exceed
the connection pool limit. If the number of free connections within a proxy connection pool
reaches zero, problems may occur.

Use the following CLI command to gather memory utilization on the device.

o get system performance status

o get system performance top <delay> <max lines> eg: “get system performance top 10 100”
o diagnose hardware sysinfo memory

3. Interface errors
If receive interface errors on Fortigate firewalls, please execute following CLI commands to
gather error information.

o diagnose hardware deviceinfo nic (This will display all interfaces on the device)
o diagnose hardware deviceinfo nic <port name> eg: diagnose hardware deviceinfo
nic Svr_VLAN
o The above information is useful to check duplex mismatch, collisions and errors.

The counters and their meaning describe what you may see when using the
CLI command diag hardware deviceinfo nic interface.
 To clear interface counters, please execute following CLI commands
o diagnose netlink interface list <port name> eg: diagnose netlink interface list
Svr_VLAN

The above is showing current interface counters on port “Svr_VLAN”

 To clear interface counters execute “diagnose netlink interface clear Svr_VLAN” whereby
“Svr_VLAN” is port name.

4. PING

The ping command sends a very small packet to the destination, and waits for a response.
The response has a timer that may expire, indicating the destination is unreachable. The
behavior of ping is very much like a sonar ping from a submarine, where the command gets
its name.

Ping is part of Layer-3 on the OSI Networking Model. Ping sends Internet Control Message
Protocol (ICMP) “echo request” packets to the destination, and listens for “echo response”
packets in reply. However, many public networks block ICMP packets because ping can be used
in a denial of service (DoS) attack (such as Ping of Death or a smurf attack), or by an attacker to
find active locations on the network. By default, FortiGate units have ping enabled while
broadcast-forward is disabled on the external interface.

What ping can tell you?

Beyond the basic connectivity information, ping can tell you the amount of packet loss (if
any), how long it takes the packet to make the round trip, and the variation in that time from
packet to packet.

If there is some packet loss detected, you should investigate the following:
 Possible ECMP, split horizon, or network loops.

 Cabling to ensure no loose connections.

 Verify which security policy was used (use the packet count column on the Policy & Objects >
Policy page).

If there is total packet loss, you should investigate the following:

 Hardware — ensure cabling is correct, and all equipment between the two locations is
accounted for.

 Addresses and routes — ensure all IP addresses and routing information along the route
is configured as expected.

 Firewalls — ensure all firewalls, including FortiGate unit security policies allow PING to pass
through.
How to use ping

Ping syntax is the same for nearly every type of system on a network.
To ping from a FortiGate unit
1. Connect to the CLI either through telnet or through the CLI widget on the web-based manager
dashboard.

2. Enter exec ping 10.11.101.101 to send 5 ping packets to the destination IP address. There are
no options for this command.

Sample output:

Head_Office_620b # exec ping 10.11.101.101

PING 10.11.101.101 (10.11.101.101): 56 data bytes

64 bytes from 10.11.101.101: icmp_seq=0 ttl=255

time=0.3 ms 64 bytes from 10.11.101.101:
icmp_seq=1 ttl=255 time=0.2 ms 64 bytes from
10.11.101.101: icmp_seq=2 ttl=255 time=0.2 ms 64
bytes from 10.11.101.101: icmp_seq=3 ttl=255
time=0.2 ms 64 bytes from 10.11.101.101:
icmp_seq=4 ttl=255 time=0.2 ms

--- 10.11.101.101 ping statistics ---

5 packets transmitted, 5 packets received, 0%

packet loss round-trip min/avg/max =
0.2/0.2/0.3 ms

To ping from an MS Windows PC

1. Open a command window.

o In Windows XP, select Start > Run, enter cmd, and select OK.
o In Windows 7, select the Start icon, enter cmd in the search box, and select cmd.exe
from the list.

2. Enter ping 10.11.101.100 to ping the default internal interface of the FortiGate unit with
four packets. Other options include:

o -t to send packets until you press “Control-C”

o a to resolve addresses to domain names where possible
o n X to send X ping packets and stop

Sample output:

C:\>ping 10.11.101.101

Pinging 10.11.101.101 with 32 bytes of data:

Reply from 10.11.101.101: bytes=32 time=10ms TTL=255

Reply from 10.11.101.101: bytes=32 time<1ms TTL=255

Reply from 10.11.101.101: bytes=32 time=1ms TTL=255

Reply from 10.11.101.101: bytes=32 time=1ms TTL=255

Ping statistics for 10.11.101.101:

Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

Approxmate round trip times in milli-seconds:

Minimum = 0ms, Maximum = 10ms, Average = 3ms

5. SNMP Uptime / Healthcheck

Simple Network Management Protocol (SNMP) is an application–layer protocol defined by the

Internet Architecture Board (IAB) in RFC1157 for exchanging management information between
network devices. It is a part of Transmission Control Protocol⁄Internet Protocol (TCP⁄IP) protocol
suite.
SNMP is one of the widely accepted protocols to manage and monitor network elements. Most
of the professional–grade network elements come with bundled SNMP agent. These agents
have to be enabled and configured to communicate with the network management system
(NMS).
If SNMP uptime or healthcheck alert triggered on Fortigate firewalls, the following pre-
assumption can be made
o Connection is down/flapping between Fortigate firewall and SNMP server
o Fortigate firewall is hard down
o Routing protocol is down

Please execute the following CLI command to verify device status.

o get system performance status (Verify uptime)
o get router info routing-table all
o get route info bgp
o diagnose debug crashlog read
 6. Host down alert

In the event of Fortigate firewall “host down alert”, please execute the following
troubleshooting steps.
o Try to access the device via telnet or SSH
o Try to PING the device from jump server
Site Location Device Name Interface Vendor Circuit ID Contact Information
Kumon Asia & Oceania SGFWRSO001 Port1 NTT Singapore 5873153000 helpdesk@ntt.com.sg
Tel: (65)-68715400
PT KIE Indonesia IDFWJKT001 Port1 NTT Indonesia n/a noc@ntt.co.id
Tel: (62)-215727777
Thailand ECL THFWNDC001 Port3 NTT Thailand AI-2015-161  support@ntt.co.th
Tel: (66)2-751-5519
Kumon Education MYFWKMY001 Ether0/0 NTT MSC Sdn Bhd NTT0494-ISP-001-947 it-sc@arc.net.my
Tel: (60)-383190000
Kumon Australia AUSWSYD001 Gi0/24 NTT Austalia W151001148-CBSWG  goc@ntt.com
Tel: 1-720-475-4200
o If the following sites firewalls down, please engage the vendors as per below listed
contacts. Custom template required to use when engaging vendors according to sites.
o Please send e-mail to local support team for following Fortigate firewalls location to
verify power & maintenance.

o For following locations, if Fortigate / Juniper firewalls are down please engage local
support team to verify the status as these sites internet connections being managed
locally.

Sites Location Device Name

Kumon Philippines PHFWMNL001
Kumon India Education INFWDEL001
Kumon Vietnam VNFWSGN001
Kumon Vietnam VNFWSGN002

o Once the device is up, please execute following CLI command to collect logs and verify
device status.
 get system performance status
 diagnose debug crashlog read