Advanced FortiGate installation and setup : Advanced troubleshooting by sniffing packets (packet capture)

Advanced troubleshooting by sniffing packets (packet capture)

Problem
How do I use filters for sniffing? They are really confusing.

Solution
You can perform some basic packet sniffing and network troubleshooting without using packet sniffing filters.
However, with filters, you can fine tune your troubleshooting to the point of being able to find a specific ping
packet on a busy network.
When packet sniffing, the filter field is very flexible. By using the filter option, you can:

match the source hostname or IP address

match the type of packet (arp, ip, gre, esp, udp, tcp, icmp)

match the port number

logically AND or OR parts of the filter with each other

specify a certain byte in a packet

The default format of the filter syntax is:

[[src|dst] host <host_name_or_IP1>] [[arp|ip|gre|esp|udp|tcp|icmp] [port_no]]
[and | or] [..]
Lets look at each of the different parts to the filter. Keep in mind that in addition to these formats, you can also
search for individual words using the filter. The following are examples.

IP matching with filters

Lets look at the hostname and IP matching [[src|dst] host <host_name_or_IP1>]. It allows you
to specify either the source or destination host. For example if you want to sniff packets coming from IP
address 192.168.1.27 you would set the filter to src host 192.168.1.27. If you want to sniff packets
going to a computer called my_laptop, the filter would be dst host my_laptop. This host name is

resolved using DNS.

In each case, when the sniffer finds packets from that computer, the packets will match the filter and be
displayed. You can enter two or more different computers using this format and join them with logical ANDs or
ORs. For example, you could specify one source and two destinations.
In the following example, lets assume a computer on the network is pinging the FortiGate unit. We will only be
looking for ping packets with a source of 172.20.120.136 which is the FortiGate unit.
diag sniffer packet any 'icmp and src host 172.20.120.136'
interfaces=[any]
filters=[icmp and src host 172.20.120.136]
0.319302 172.20.120.136 -> 172.20.120.17: icmp: echo reply
1.348780 172.20.120.136 -> 172.20.120.17: icmp: echo reply
2.355177 172.20.120.136 -> 172.20.120.17: icmp: echo reply
3.356008 172.20.120.136 -> 172.20.120.17: icmp: echo reply
4 packets received by filter
0 packets dropped by kernel
The result displays four packets, all ping (icmp) packets, originating from the FortiGate unit and going to
172.20.120.17. This time there was no verbosity level indicated or number of packets. A default verbosity level
1 is used, and the sniffing continues until you press Ctrl-C to stop it. Note that the last two lines tell you how
many packets were sniffed and if the FortiGate kernel dropped any packets during this time.
When the sniffing has ended, if you see anything but zero packets dropped, you may
have a problem. Packets dropped indicates the FortiGate unit was not able to sniff and
display all the packets that were coming in. If you were looking for all the packets in a
sequence, there may well be packets missing. For this reason, you should consider
possible reasons for those dropped packets, attempt to fix the problem so all packets
are captured, and run the sniffer again. Keep in mind that the sniffer can take up to
25% of the CPU resources on smaller FortiGate units.

Sniffing a port and specifying multiple hosts using AND and OR operators
When a TCP session is created, the destination port is set to a known port number for example, port 80 is
commonly used for HTTP sessions. But the source port is randomly assigned. The unknown source port can
make troubleshooting difficult. However, the FortiGate packet sniffer can match the known port if it is the
source or destination port you do not need to know which port.
Lets check HTTP packets going between IP 172.20.120.18 (the FortiGate) and on either 10.10.80.110 (wifi
interface called Star) or 10.10.10.100 (internal LAN interface).
diag sniffer packet any "port 80 and host 172.20.120.18 and (host 10.10.80.110 or host 10.10.10.100)" 4

interfaces=[any]
filters=[port 80 and host 172.20.120.18 and (host 10.10.10.100 or host
10.10.80.110)]
5.036340 internal in 10.10.10.100.58753 -> 172.20.120.18.80: syn 4189154
5.036664 internal out 172.20.120.18.80 -> 10.10.10.100.58753: syn 1354149395
ack 4189155
6.464015 Star out 172.20.120.18.80 -> 10.10.80.110.56791: syn 2000204115 ack
571678006
6.471966 Star in 10.10.80.110.56791 -> 172.20.120.18.80: ack 2000204116
6.474720 Star in 10.10.80.110.56791 -> 172.20.120.18.80: psh 571678006 ack
2000204116
5.036837 internal in 10.10.10.100.58753 -> 172.20.120.18.80: ack 1354149396
5.037023 internal in 10.10.10.100.58753 -> 172.20.120.18.80: psh 4189155 ack
1354149396
6.463686 Star in 10.10.80.110.56791 -> 172.20.120.18.80: syn 571678005
Since either the source or destination will be using port 80, all HTML traffic between those two computers will

match the filter and be displayed. SSH and HTTPS traffic uses different ports, so that traffic will not be
displayed. The first number of each line of output will vary between sources and is a good way to quickly
determine which IP addresses are in that session.

Packet type filters

Lets look at the packet type [arp|ip|gre|esp|udp|tcp]. This determines what type of packets to look
for. In addition to the common ICMP, IP, TCP, and UDP you can look for ARP (address resolution protocol),
GRE (generic routing encapsulation), and ESP (encapsulating security payload) packets.
If the protocol you want isnt listed here you can specify it if you know the ethernet
protocol number for it. For example to specify ARP packets on the internal interface
with this method: diag packet sniffer internal ether proto 0x0806
Lets sniff some ARP packets from a gateway on the network at IP address 172.20.120.2. For this we dont
care about the interface, and five packets will be enough to see what is happening.
# diag sniffer packet any 'arp' 1 5
interfaces=[any]
filters=[arp]
1.187291 arp who-has 192.168.100.1 tell 192.168.100.99
2.187125 arp who-has 192.168.100.1 tell 192.168.100.99
2.858334 arp who-has 172.20.120.228 tell 172.20.120.224
2.889542 arp who-has 172.20.120.224 tell 172.20.120.228
4.187019 arp who-has 192.168.100.1 tell 192.168.100.99
From this output, we can see ARP requests from a computer with IP address 192.168.100.99 that is looking
for the MAC address of a computer with the IP address 192.168.100.1. In the ARP protocol, the who-has
request is broadcast and includes the link layer address of where to send the reply. The expected response,
when a computer has the 192.168.100.1 IP address, will be in the format arp reply 192.168.100.1 is
at 00:26:b9:00:0f:9c. Since there is no such reply in the sniffed packets, we can either sniff more
packets or assume there is no computer on the network with the IP address 192.168.100.1. This may be
important if a computer is supposed to be using that IP address and is not. It could imply DHCP problems, or
that the computer was physically moved to a different part of the network.
ARP packets can be the source of problems if there is a network loop. As mentioned
above, ARP tries to match a single MAC address to a single IP address. If the request
results in two or more replies with the same IP address, or different IP addresses have
the same MAC address, as may happen with virtual networking solutions, the loop or
asymmetric routing is created. Essentially, all traffic will go to and from both computers.
This will appear as a network slowdown or halt. You can see this happening if you are
sniffing ARP packets and seeing the double replies or double MAC addresses. To
confirm that this is the issue, enter the CLI command config system settings,
set asymroute enable, end. This will turn on asymmetric routing, stop these
ARP problems, and disable stateful inspection. Disabling stateful inspection will
compromise security, so in most cases you should only use this command to confirm a
problem. Once the problem is confirmed, use the sniffer output to find and fix the
source and then disable asymmetric routing.

Miscellaneous advanced filters

There are some non-standard filters you can use to match traffic with the packet sniffer. These advanced
filters use logical symbols to match specific bits within packet headers. Some examples are:
If you want to match TTL = 1 in the packet headers on port2:
# diagnose sniffer packet port2 ip[8:1] = 0x01
If you want to match packets with a source IP address of 192.168.1.2 in the header:

# diagnose sniffer packet internal "(ether[26:4]=0xc0a80102)"

The source and destination information are stored in different places in the packet headers. If you want to
match packets with a source MAC address of 00:09:0f:89:10:ea on the internal interface
# diagnose sniffer packet internal "(ether[6:4]=0x00090f89) and
(ether[10:2]=0x10ea)"
where matching packets with the same MAC address as a destination MAC on the internal interface is
# diagnose sniffer packet internal "(ether[0:4]=0x00090f89) and
(ether[4:2]=0x10ea)"
You can also target specific types of packets, such as addressing the TCP or UDP flags.
If you want to match packets with RST flag set:
# diagnose sniffer packet internal "tcp[13] & 4 != 0"
If you want to match packets with the SYN flag set:
# diagnose sniffer packet internal "tcp[13] & 2 != 0"
If you want to match packets with the SYN-ACK flag set:
# diagnose sniffer packet internal "tcp[13] = 18"

If your FortiGate unit has NP2 interfaces that are offloading traffic, this will change the
sniffer trace. Before performing a trace on any NP2 interfaces, you should disable
offloading on those interfaces.

Best practices
Here are some tips that will improve your troubleshooting using the packet sniffer.

Enabling the sniffer will consume additional CPU resources. This can be as high as an additional 25

percent of CPU usage on low-end models. Therefore, enabling this on a unit that is experiencing
excessively high CPU usage, can only render the situation worse. If you must perform a sniff, keep the
sniffing sessions short and keep the filter specific.

Try to always include ICMP in the sniffer filter. You may capture an ICMP error message that can help
identify the cause of the problem. For example:
diag sniff packet interface wan1 'tcp port 3389 or icmp' 3

Use the any interface to sniff all FortiGate unit interfaces. You can use the "any" interface if you want to
confirm that a specific packet is sent and received by different FortiGate interfaces. The any interface is
also useful if you are not sure which interface will send or receive the packet. An example using the any
interface:
diag sniff packet any 'tcp port 3389' 3

The FortiGate unit may not display all packets if too much information is requested. When this occurs, the
FortiGate unit will log the following message once the trace is terminated:
12151 packets received by filter
3264 packets dropped by kernel
When this occurs, it is possible that what you were attempting to capture, was not actually captured. In
order to avoid this, try to make the filters more specific, reduce the verbosity level, or run the sniffer
during a lower traffic period.

The packet timestamps, as displayed by the sniffer, may become skewed or delayed under high load
conditions. This may occur even if no packets were dropped. Therefore, it is not recommended that you
rely on these values in order to troubleshoot or measure performance issues that require absolute precise
timing.

Short Ethernet frames sent by the FortiGate unit may appear to be under the minimum length of 64 bytes
(also known as runts) and will not be displayed by the sniffer. This is because the sniffer does not display
any Ethernet Trailer/Padding information, although it is sent over the network.

The Ethernet source and/or destination MAC addresses may be incorrect when using the "any" interface.
They may be displayed as all zeros (00:00:00:00:00:00) or 00:00:00:00:00:01.

Try to always include ICMP in the sniffer filter. You may capture an ICMP error message that can help
identify the cause of the problem. For example, diag sniff packet interface wan1 'tcp port
3389 or icmp' 3

If you are sniffing VLAN packets, you cannot have any filter configured if you want to see the VLAN tags.
For example diag sniffer packet wan1 icmp will not show the tags where diag sniffer
packet wan1 will.