Troubleshooting Tip: Using the FortiOS built-in pa... - Fortinet Community https://community.fortinet.com/t5/FortiGate/Troubleshooting-Tip-Using...

 Help 

Forums  Knowledge Base  Community Groups  Blogs

FortiGate
FortiGate Next Generation Firewall utilizes purpose-built security
processors and threat intelligence security services from FortiGuard
labs to deliver top-rated protection and high performance, including
encrypted traffic.

This Board Search here

Fortinet Community  Knowledge Base  FortiGate  Troubleshooting Tip: Using the FortiOS built-in pa...

pkungatti_FTNT
Staff

Created on Edited on By

08�26�2005 12�00 AM 12�22�2022 01�05 AM Jean-Philippe_P

Troubleshooting Tip: Using the FortiOS built-in packet sniffer

for capturing packets
Description

This article describes one of the troubleshooting options available in FortiGate CLI to
check the traffic flow, by capturing packets reaching the FortiGate unit.

The CLI offers in addition to the GUI packet capture methods the possibility to capture on
multiple interfaces and mark these per packets. This can be useful if suspecting a packet
leaving on the wrong interface and being dropped by FortiGate.

1 of 13 2/3/2023, 12:10 PM
Troubleshooting Tip: Using the FortiOS built-in pa... - Fortinet Community https://community.fortinet.com/t5/FortiGate/Troubleshooting-Tip-Using...

Scope

FortiGate, all FortiOS versions.

other Fortinet appliances providing a CLI sniffer: FortiAnalyzer - FortiMail - FortiManager.

Solution

Content: Packet sniffer - basics.

Basic packet-capture commands:

- Example 1: Simple trace.
- Example 2: Simple trace.

Filter Functionality:
- Example 3: Trace with filters.
- Uncommon scenarios.

Sniffer Basics:

The packet sniffer 'sits' in the FortiGate and can display the traffic on a specific interface
or on all interfaces.

There are three different levels of Information, also known as Verbose Levels 1 to 3,
where verbose 1 shows less information and verbose 3 shows the most.
Verbose Levels 4, 5, and 6 would additionally provide the interface details.

Verbose levels in detail:

1: print header of packets.

2: print header and data from IP of packets.
3: print header and data from Ethernet of packets.
4: print header of packets with interface name.
5: print header and data from IP of packets with interface name.
6: print header and data from Ethernet of packets with interface name.

2 of 13 2/3/2023, 12:10 PM
Troubleshooting Tip: Using the FortiOS built-in pa... - Fortinet Community https://community.fortinet.com/t5/FortiGate/Troubleshooting-Tip-Using...

This article walks through some examples and different levels of verbosity to show the
different possibilities for debugging.

Basic sniffing command:

All packet sniffing (packet capture) commands start like this:

# diag sniffer packet <interface> <'filter'> <verbose> <count> a

<interface> can be an interface name or "any" for all interfaces
<'filter'> is a very powerful filter functionality which will be described in more detail
<verbose> means the level of verbosity as described already
<count> the number of packets the sniffer reads before stopping.
a – timestamps the packets with the absolute UTC time
l - (small letter L) timestamps the packets with LOCAL time on the unit
(blank/no letter) – relative to the beginning of the capture

Note: for parallel captures on multiple interfaces/SSH sessions on FortiGate, use 'a' or 'l',
do not leave it blank.

Note: in certain cases, where the unit has the capability and the session can be handled
by a dedicated processor, the session is offloaded from the kernel, making it impossible
to capture these packets. In this case, turn off the offloading in the policy that matches
the traffic with 'set auto-asic-offload disable' for troubleshooting purposes only, and
revert to the initial state after the capture.

Example 1: Simple sniffer.

Sniff 3 packets of all traffic with verbose Level 4 on wan1 Interface.

# diag sniffer packet wan1 none 1 3

interfaces=[wan1]
filters=[none]
0.996031 arp who-has 10.109.19.70 tell 10.109.31.254
1.310085 10.109.16.137.80 -> 172.26.48.21.56361: fin 3133701817 ack
1838214099
1.310648 172.26.48.21.56361 -> 10.109.16.137.80: ack 3133701818

There is an ARP packet and a TCP session tear-down finished.

Because the 10.109.16.137 IP Address uses Port 80 (10.109.16.137.80) it is possible to
assume that some packets have been caught from a running HTTP session.

The 'none' variable means 'no filter applied', '1' means 'verbose 1' and '3' means 'catch 3
packets and stop'.

Note that there is no timestamp switch at the end, therefore the first packet was
received after 0.996031 seconds since the command was issued.

Example 2: Simple Trace.

3 of 13 2/3/2023, 12:10 PM
Troubleshooting Tip: Using the FortiOS built-in pa... - Fortinet Community https://community.fortinet.com/t5/FortiGate/Troubleshooting-Tip-Using...

Sniff 3 packets of all traffic with verbose Level 4 on the wan1 interface.

# diag sniffer packet wan1 none 4 3

0.455401 wan1 -- 172.26.48.21.55992 -> 10.109.16.137.80: syn
1906363376
0.455490 wan1 -- 10.109.16.137.80 -> 172.26.48.21.55992: syn
1423685325 ack 1906363377
0.456892 wan1 -- 172.26.48.21.55993 -> 10.109.16.137.80: ack
123332141

There is some more interesting information, just when a TCP session was being set up
(TCP handshake).

172.26.48.21 tries to connect to 10.109.16.137 on Port 80 with a SYN and gets a SYN
ACK back.
Finally, the session is acknowledged and established after the 3-way TCP handshake.

With the information level set to Verbose 4, additionally summary of the Source and
Destination IP Addresses are visible.

If there is no <count> value (or count=0), the Sniffer runs forever until you stop it with
<CTRL-C>.

Hint: For further investigation, it is always a good idea to log the SSH output to a file. If
Putty is used (a free SSH client for Windows) it is possible to easily log all output to a file
which to search/sort/process.

Verbose 5 and Verbose 6 levels:

Verbose 5 contains much more information

The IP Header as already seen in Verbose 4.
The Payload of the IP packet itself.

An output of Verbose 5 looks like this:

# diag sniffer packet any none 5 1

interfaces=[any]
filters=[none]
0.529129 wan1 in 10.109.19.165.137 -> 10.109.31.255.137: udp 50
0x0000 4500 004e 153b 0000 8011 dce6 0a6d 13a5
E..N.;.......m..
0x0010 0a6d 1fff 0089 0089 003a a7b6 8db5 0110
.m.......:......
0x0020 0001 0000 0000 0000 2045 4745 5046 4346
.........EGEPFCF
0x0030 4545 4a45 4f45 4646 4543 4e46 4145 4443
EEJEOEFFECNFAEDC

4 of 13 2/3/2023, 12:10 PM
Troubleshooting Tip: Using the FortiOS built-in pa... - Fortinet Community https://community.fortinet.com/t5/FortiGate/Troubleshooting-Tip-Using...

0x0040 4143 4143 4143 4143 4100 0020 0001 ACACACACA.....

Notice the in/out parameter after the wan1 interface that will confirm the direction of
the packet entering or leaving the interface.

Verbose 6, finally, even includes Ethernet (Ether Frame) Information.

A script is available (fgt2eth.pl), which will convert a captured verbose 6 output, into a
file that can be read and decoded by Ethereal/Wireshark. See the end of this article for
details.

Use of absolute time stamp in sniffer trace will report the absolute system time (no time
zone) in packet summary:

# diag sniffer packet wan1 none 4 2 a

interfaces=[wan1]
filters=[none]
2019-08-16 09:36:02.570320 wan1 -- arp who-has 10.109.16.153 tell
10.109.16.152
2019-08-16 09:36:02.663102 wan1 -- 172.26.48.21.64241 ->
10.109.16.137.80: fin 2427687875 ack 3609408424

Hint: Below is the format that Technical Support will usually request when attempting to
analyze a problem as it includes full packet content, as well as absolute time stamp, in
order to correlate packets with other system events.

# diag sniffer packet any <'filter'> 6 0 a

Filter functionality:

As already mentioned, diag sniffer includes a powerful filter functionality that will be
described here.

FortiOS gives a brief example:

# diag sniffer packet any ?

<filter> Flexible logical filters for sniffer (or "none").
For example: To print UDP 1812 traffic between forti1 and either
forti2 or forti3
'udp and port 1812 and host forti1 and ( forti2 or forti3 )'

If a second host is specified, only the traffic between the 2 hosts will be displayed.

Imagine capturing the traffic from one PC to another PC.

Without a filter, the sniffer will display all packets, which is far too much data and quite
painful to sort and debug a large file.

Example 3: Trace with Filters:

To see what's going on between two PCs (or a PC and a FortiGate), do not forget the

5 of 13 2/3/2023, 12:10 PM
Troubleshooting Tip: Using the FortiOS built-in pa... - Fortinet Community https://community.fortinet.com/t5/FortiGate/Troubleshooting-Tip-Using...

quotes delimiting the filter expressions:

# diag sniffer packet wan1 'src host 10.109.16.137 and dst host
172.26.48.21' 1 3
interfaces=[wan1]
filters=[src host 10.109.16.137 and dst host 172.26.48.21]
1.453488 10.109.16.137.80 -> 172.26.48.21.61776: syn 3263501252 ack
507821611
1.454138 10.109.16.137.80 -> 172.26.48.21.61776: ack 507822560
1.457612 10.109.16.137 -> 172.26.48.21: icmp: echo reply

Assuming there is a lot of traffic on the wire, this filter command will only display traffic
(but all traffic) from Source 10.109.16.137 to Destination 172.26.48.21.
It will NOT show traffic to 10.109.16.137 (for example the ICMP reply) because it is
asked for 'src host' and 'dst host'.

Note.
When 'src' and 'dst' are used, 'host' word is optional, and is applied by default.
It is also possible to use 'net' as keyword for a broader result:

# diag sniffer packet wan1 'src 10.109.16.137 and net 172.26.48.0/20' 1

3

However, when filtering for bidirectional traffic, either use 'host' or CIDR notated 'host'
arguments:

# diag sniffer packet wan1 'host 10.109.16.137 and host 172.26.48.0/20'

1 3

To have only a specific type of traffic (for example TCP Traffic only) it is necessary to
change the filter slightly:

# diag sniffer packet wan1 'src 10.109.16.137 and tcp' 1 3

interfaces=[wan1]
filters=[src 10.109.16.137 and tcp]
1.729308 10.109.16.137.80 -> 172.26.48.21.62137: ack 4024820721
1.729541 10.109.16.137.80 -> 172.26.48.21.62139: syn 3670899298 ack
1522348774
1.730308 10.109.16.137.80 -> 172.26.48.21.62139: ack 1522349665

Though ICMP (ping) was also running and probably some DNS requests too, the trace
only shows the TCP part.
The Source is: 10.109.16.137.80 which is IP 10.109.16.137 on Port 80. Apparently, there
is an HTTP session to 10.109.16.137.

The same the other way around (using here ‘host’, it shows the traffic both ways):

# diag sniffer packet any 'host 10.109.16.137 and host 172.26.48.21' 1

6 of 13 2/3/2023, 12:10 PM
Troubleshooting Tip: Using the FortiOS built-in pa... - Fortinet Community https://community.fortinet.com/t5/FortiGate/Troubleshooting-Tip-Using...

5
interfaces=[any]
filters=[host 10.109.16.137 and host 172.26.48.21]
1.182532 172.26.48.21.55585 -> 10.109.16.137.80: syn 3194317969
1.182598 10.109.16.137.80 -> 172.26.48.21.55585: syn 2863972551 ack
3194317970
1.183166 172.26.48.21.55585 -> 10.109.16.137.80: ack 2863972552
1.183360 172.26.48.21.55585 -> 10.109.16.137.80: psh 3194317970 ack
2863972552
1.183406 10.109.16.137.80 -> 172.26.48.21.55585: ack 3194318935

In this example, it is sniffing for ICMP only, to and from 10.109.16.137:

# diag sniffer packet any 'host 10.109.16.137 and icmp' 1 30

interfaces=[any]
filters=[host 10.109.16.137 and icmp]
16.866489 172.26.48.21 -> 10.109.16.137: icmp: echo request
16.866581 10.109.16.137 -> 172.26.48.21: icmp: echo reply

Another useful feature is a logical combination.

Let's assume it is necessary to check for ICMP and TCP only (but not for UDP, ARP, etc).

It is possible to combine protocols in the following manner.

This sniff will display all TCP or ICMP traffic to and from host 10.109.16.137, in the
verbose 1 level.

# diag sniffer packet wan1 'host 10.109.16.137 and (icmp or tcp)' 1

interfaces=[wan1]
filters=[host 10.109.16.137 and (icmp or tcp)]
0.748627 172.26.48.21 -> 10.109.16.137: icmp: echo request
0.748681 10.109.16.137 -> 172.26.48.21: icmp: echo reply
1.138054 172.26.48.21.55741 -> 10.109.16.137.80: fin 1192893176 ack
3520420646
1.138313 172.26.48.21.55742 -> 10.109.16.137.80: syn 3309427492

It is now necessary to further limit the sniffer filter.

Sniff traffic between 2 hosts, but only TCP and only port 80.

# diag sniffer packet wan1 'host 10.109.16.137 and host 172.26.48.21

and tcp port 80' 1 3
interfaces=[wan1]
filters=[host 10.109.16.137 and host 172.26.48.21 and tcp port 80]
2.120582 10.109.16.137.80 -> 172.26.48.21.55816: fin 430415930 ack
3124927915
2.120994 172.26.48.21.55816 -> 10.109.16.137.80: ack 430415931
2.124248 172.26.48.21.55816 -> 10.109.16.137.80: fin 3124927915 ack
430415931

7 of 13 2/3/2023, 12:10 PM
Troubleshooting Tip: Using the FortiOS built-in pa... - Fortinet Community https://community.fortinet.com/t5/FortiGate/Troubleshooting-Tip-Using...

A logical 'AND' is used in this command between 10.109.16.137 and 172.26.48.21 such
that only packets containing both these host addresses will be seen.
Even if telnet and ssh traffic was transferred between the two hosts, it only shows port
80 TCP traffic.

Uncommon scenarios:

Filters can also be used to display packets based on their content, using a hexadecimal
byte position.

Match TTL = 1
diagnose sniffer packet any "ip[8:1] = 0x01"

Match Source IP address = 192.168.1.2:

# diagnose sniffer packet wan1 "(ether[26:4]=0xc0a80102)"

Match Source MAC = 00:09:0f:89:10:ea:

# diagnose sniffer packet any "(ether[6:4]=0x00090f89) and

(ether[10:2]=0x10ea)"

Match Destination MAC = 00:09:0f:89:10:ea:

# diagnose sniffer packet any "(ether[0:4]=0x00090f89) and

(ether[4:2]=0x10ea)

Match ARP packets only:

# diagnose sniffer packet wan1 "ether proto 0x0806"

TCP or UDP flags can be addressed using the following:

Match packets with RST flag set:

# diagnose sniffer packet wan1 "tcp[13] & 4 != 0"

Match packets with SYN flag set:

# diagnose sniffer packet wan1 "tcp[13] & 2 != 0"

Match packets with SYN-ACK flag set:

# diagnose sniffer packet wan1 "tcp[13] = 18"

Match HA heartbeat packets only:

# diagnose sniffer packet ha1 "ether proto 0x8890"

8 of 13 2/3/2023, 12:10 PM
Troubleshooting Tip: Using the FortiOS built-in pa... - Fortinet Community https://community.fortinet.com/t5/FortiGate/Troubleshooting-Tip-Using...

Related article for HA heartbeat packet troubleshooting:

https://community.fortinet.com/t5/FortiGate/Technical-Tip-FortiGate-HA-Heartbeat-
packet-Ethertypes/t...

Sniffer for VLAN traffic:

This is a common scenario, where the VLAN interface is used instead of the interface
name in the command line.
For example, sniffing the traffic for host 11.11.11.9 in the VLAN interface 'vlan206', the
command would be:

# diag sniffer packet any "host 11.11.11.9" 4 0

interfaces=[any]
filters=[host 11.11.11.9]
1.774584 vlan206 in 11.11.11.9 -> 11.11.11.1: icmp: echo request
1.774642 vlan206 out 11.11.11.1 -> 11.11.11.9: icmp: echo reply
1.774648 dmz out 11.11.11.1 -> 11.11.11.9: icmp: echo reply

# diag sniffer packet vlan206 "host 11.11.11.9" 4 0

interfaces=[vlan206]
filters=[host 11.11.11.9]
0.968800 vlan206 -- 11.11.11.9 -> 11.11.11.1: icmp: echo request
0.968858 vlan206 -- 11.11.11.1 -> 11.11.11.9: icmp: echo reply
1.982626 vlan206 -- 11.11.11.9 -> 11.11.11.1: icmp: echo request
1.982683 vlan206 -- 11.11.11.1 -> 11.11.11.9: icmp: echo reply

Note that when the vlan206 interface is used as a filter, the underlying physical interface
is not shown in the capture.

In some cases, when sniffing traffic for host address by default underlying physical
interface is not displayed for incoming traffic, however, associated VLANs and physical
interface for outbound traffic are displayed.

For example:

From the above capture, it is possible to see DMZ is a physical interface and vlan206 is
an associated VLAN when traffic ingress DMZ is not visible in, and only VLAN 206 in, but
when traffic goes out we see VLAN 206 out and DMZ out.

This is because when a filter with host x.x.x.x is set in sniffer, FortiGate has to strip out
the VLAN ID and frames first to know the host address to capture the traffic, hence it is
not sure if the traffic is coming via the DMZ interface.

However, for outgoing traffic, FortiGate will retag the packets, so it knows it goes out via
the DMZ interface.

If the sniffer filter is changed like below, a DMZ should be visible in the interface as well
for incoming packets.

9 of 13 2/3/2023, 12:10 PM
Troubleshooting Tip: Using the FortiOS built-in pa... - Fortinet Community https://community.fortinet.com/t5/FortiGate/Troubleshooting-Tip-Using...

# diag sniffer packet any ' ' 4

Also, when capturing the traffic with Verbose 6 (to see the contents of the packet), the
VLAN tag is stripped when the filter is run on the VLAN interface or 'any' interface.

To capture the VLAN tag in a packet capture, the sniffer must be run on the underlying
physical interface.

In this situation, the filters cannot be used:

# diag sniffer packet dmz none 6 0

interfaces=[dmz]
filters=[none]
pcap_lookupnet: dmz: no IPv4 address assigned
0.981174 dmz -- 802.1Q vlan#206 P0
0x0000 d4be d99c 02f9 085b 0e37 ef37 8100 00ce .......
[.7.7....
0x0010 0800 4500 0040 3f4a 4000 7406 d33f d992
..E..@?J@.t..?..
0x0020 0488 0b0b 0b09 0050 c131 ab2a 473b 1fb3
.......P.1.*G;..
0x0030 bc14 5018 0401 8da5 0000 1130 1b00 0000
..P........0....
0x0040 0000 1f00 0000 1c00 0000 1b00 0000 1800
................
0x0050 0000 ..

1) Also attached is the fgt2eth.pl script that will convert a verbose level 3 or 6 sniffer
output, into a file readable and decodable by Ethereal/Wireshark PCAP file. In case the
traffic is sniffed without an interface filter ('diagnose sniffer packet any ''6 0 a'), by
default the script will create a single file with traffic sniffed on all interfaces. However, it
is possible to create a per-interface PCAP file by adding a '-demux' argument while
converting text to pcap file (pcap.exe -in <input file> -out <output file> -demux).

2) The fgt2eth.exe file is also attached to this article, this file is outdated and is not
supported but may provide some guidance.

3) The attached scripts are provided 'as is' and are not supported by Technical Support.

Another conversion script that provides similar functionality is available for download
here. The above terms apply to this script as well. In addition to 'fgt2eth.pl', sniftran also
adds to the pcap file the interface labels as comments for each packet, making it easier
to identify incoming/outgoing traffic.

Related Articles:

10 of 13 2/3/2023, 12:10 PM
Troubleshooting Tip: Using the FortiOS built-in pa... - Fortinet Community https://community.fortinet.com/t5/FortiGate/Troubleshooting-Tip-Using...

Troubleshooting Tip : First steps to troubleshoot connectivity problems to or through a

FortiGate wi...
Troubleshooting Tip: Using the FortiGate sniffer on VLAN interfaces
Troubleshooting Tip: Packet capture (CLI sniffer) tips and best practices
Technical Note : How To Troubleshoot Wireless Station Connection Issues on the FortiAP
Technical Note: Packet capture buffer limit

 fgt2eth.exe 
explanation_on_how_to_packet_capture_for_only_certain_TCP_flags_v2.txt
 
10 KB
fgt2eth.pl
 
12 KB

FortiGate Packet Capture Packet Sniffer Sniffer

595629 4

Share

Contributors

 pkungatti_FTNT

AlexC�FTNT

Madhu_G

Anthony_E

11 of 13 2/3/2023, 12:10 PM
Troubleshooting Tip: Using the FortiOS built-in pa... - Fortinet Community https://community.fortinet.com/t5/FortiGate/Troubleshooting-Tip-Using...

Markus_M

abarushka

Jean-Philippe_P

Anonymous

Broad. Integrated. Automated.

The Fortinet Security Fabric brings together the concepts of convergence and
consolidation to provide comprehensive cybersecurity protection for all users, devices,
and applications and across all network edges.

Social Media

SECURITY RESEARCH

Threat Research

FortiGuard Labs

Threat Map

Threat Briefs

Ransomware

COMPANY

12 of 13 2/3/2023, 12:10 PM
Troubleshooting Tip: Using the FortiOS built-in pa... - Fortinet Community https://community.fortinet.com/t5/FortiGate/Troubleshooting-Tip-Using...

About Us

Security Fabric

Exec. Mgmt

Careers

Certifications

Events

Industry Awards

Social Responsibility

NEWS & ARTICLES

News Releases

News Articles

Trademarks

CONTACT US

Corporate

Community

Copyright 2023 Fortinet, Inc. All Rights Reserved.

Terms of Service Privacy Policy GDPR

13 of 13 2/3/2023, 12:10 PM