Hello and welcome to this Sophos Certified Technician training course for Sophos XG

Firewall. This is Module 801: Troubleshooting Tools.

Sophos Certified Technician

Sophos XG Firewall S801: Troubleshooting Tools

July 2017
Product version: Sophos XG Firewall v16.0

© 2017 Sophos Limited. All rights reserved. No part of this document may be used or
reproduced in any form or by any means without the prior written consent of Sophos.

Sophos and the Sophos logo are registered trademarks of Sophos Limited. Other
names, logos and marks mentioned in this document may be the trademarks or
registered trademarks of Sophos Limited or their respective owners.

While reasonable care has been taken in the preparation of this document, Sophos
makes no warranties, conditions or representations (whether express or implied) as
to its completeness or accuracy. This document is subject to change at any time
without notice.

Sophos Limited is a company registered in England number 2096520, whose

1
registered office is at The Pentagon, Abingdon Science Park, Abingdon, Oxfordshire,
OX14 3YP.

1
Once you complete this module you will be able to:
 Understand the troubleshooting process
 Understand how to access Sophos XG Firewall to perform troubleshooting
 Identify key troubleshooting tools
 Understand how each tool is used and when it should be used
 Understand High Availability
 Troubleshoot High Availability

2
Let’s start by looking at the recommended steps for troubleshooting Sophos XG
Firewall.

When you experience an issue, or an issue is reported to you, we recommend that

you firstly define the issue. To do this, you will need to confirm the symptom(s) of the
issue and determine whether an error message is being returned. You would also
need to confirm the steps to reproduce the error or symptom being reported.

Once you have identified the symptom or confirmed the error returned, you will need
to determine what is causing the issue, we refer to this as root cause analysis. During
this step you will check to see if there is any information about the error or symptom
using the Sophos knowledgebase. You will also check the log files and configuration
for information regarding the error or symptom. You will also make use of the
troubleshooting tools available. We will cover these in detail throughout this module.

Once you have determined the probable root cause, you will then perform
verification testing. To do this, you will complete the resolution steps and then
confirm if these steps have resolved the issue.

If you are unable to identify the root cause of an issue and resolve it we recommend
that you contact Sophos Support with a detailed description of the issue, including
replication steps and any errors along with the root cause analysis steps you have

3
taken.

Throughout this course you will see the troubleshooting process steps indicated for
each issue discussed using the 1, 2 and 3 progress icons as shown on screen.

3
Many of the troubleshooting steps we will be covering in this course will require shell
access to the Sophos XG Firewall.

Shell access must be enabled in the WebAdmin Console. You will go to System >
Administration > Device Access in the Local Service ACL section under the Admin
Services section. Ensure that the check box for SSH is enabled for the required zones
for example LAN and WAN. We recommend not enabling SSH for the whole WAN
zone, instead, create a service ACL Rule to allow specific IPs.

SSH access with public key authentication can also be configured. This provides
secure access to the device for multiple users without the risk of password
management. This is configured in System > Administration > Device Access.

Sophos XG Firewall supports RSA, DSA and ECDSA keys of 1024, 2048 and 4096 bits in
length. Keys can be created using a tool such as PuTTY Key Generator on Windows, or
ssh-keygen on Linux.

SSH connections are logged in /log/dropbear.log. When the SSH connection is

authenticated using keys, the thumbprint of the key is logged with the IP address that
the connection was initiated from.

4
To access Sophos XG Firewall use an SSH client (like PuTTY) to connect to the XG
Firewall Command Line Interface (CLI).

Use the Sophos XG Firewall IP address to establish the connection. Enter the
username and password of the admin account. Please note that ONLY the built-in
admin account can be used to access Sophos XG Firewall.

Once logged in you will have access to the Command Line Interface (CLI). This is
where you will perform the majority of your troubleshooting steps when supporting
Sophos XG Firewall.

5
Now you have shell access and have connected using your chosen client you will see
the command line interface (CLI) main menu.

Selecting option 4. Device Console will allow you run CLI commands. These are
Sophos XG Firewall specific commands.

6
Advanced Shell Commands are Linux operating system commands. The Advanced
Shell is accessed by selecting option 5: Device Management from the Main Menu
and then option 3: Advanced Shell.

Handy Tips:
• If you are in the Advanced Shell you can switch back to the CLI by typing cish.
• To return to the Main Menu either type exit or use 0 where applicable.

7
You will also use the Sophos XG Firewall User Interface (GUI) to troubleshoot various
issues.

You have access to useful troubleshooting tools in MONITOR & ANALYZE >
Diagnostics. As discussed in the Engineer course you can find the Log Viewer in the
top right hand corner of almost every screen in WebAdmin. The Log Viewer can be
used to view logs regarding specific functions for example Application Filter or
Authentication.

8
Logs can also be checked using the Advanced Shell. Logs are written to the /log
directory. The log files will contain a lot of information concerning the Sophos XG
Firewall and therefore you will need to know which log you are looking for and what
information you require from that log file.

Handy Tips:
• Use ls to list all logs available.
• Use pwd (present working directory) to confirm the directory you are currently in.

9
tail –f /log/<logfile-name.log> shows the current log file entries, any
new entries will be written to the log file and shown as they happen.

10
less /log/<logfile-name.log> shows the log file as a static file. Less
allows you to easily scroll through the file entries and can be used to match
keywords.

11
grep <keyword> /log/<logfile-name.log> applies a search filter
within the log file.

12
When an issue is reported, it is recommended to use the troubleshooting process
outlined previously in this module. Define the issue, perform root cause analysis and
carry out verification testing to confirm the issue is resolved.

For the root cause analysis process, there are a number of troubleshooting tools
available. Each tool should be used depending on the type of issue being reported.
We will discuss each of the available tools and when they would be used.

Later in the course, we will look at examples of issues and show the tools being used.

13
Verification tools are commands that can be used in the Command Line Interface
(CLI) to obtain specific information about the XG Firewall.

Service status can be used to show the list of services currently running. To obtain the
service status run the Advanced Shell command service –S. The S used must be a
capital for this command to be successful. The output will display the service name
and it’s status;
• RUNNING = the service is running
• UNREGISTERED = the service is not configured
• UNTOUCHED = the service is not in use
If a service is running in debug mode it will show as RUNNING, debug.

The Advanced Shell can also be used to start, stop, restart and set a service to debug.
The command used is service <service name>:start –ds nosync.
Substitute start with stop, restart, status or debug to change the function of the
command as required. Note that not all services have the option to debug.

14
Version Info is used to verify the firmware build version and unique device ID of the
Sophos XG Firewall.

The CLI command to use is system diagnostics show version. The

output displays all the database versions of individual proxies and device IDs. It is
useful for troubleshooting issues regarding licensing.

15
The interfaces command is used to show all interfaces on the Sophos XG Firewall and
to verify the IP address, subnet mask and gateway details. The command used is
ifconfig –a which will display all interfaces.

The advanced shell command ifconfig <interface label> is used to

verify the details for a specific interface.

These commands are useful when troubleshooting issues with gateway or interface
connectivity or slow internet browsing. The output will show the total incoming and
outgoing packets along with dropped packets and number of collisions.

16
The Ethtool command will display the settings for a specific interface.

This command should be used to troubleshoot issues like interface connectivity,

speed issues and slow internet browsing. The advanced shell command to use is
ethtool <interface label>.

The output will differ depending on the model of Sophos XG Firewall (Hardware or
virtual) that is installed. The tool uses specific drivers that may not be required with a
virtual installation of Sophos XG Firewall. If the drivers exist the output will include
layer 2 information including speed duplex, Port type, auto-negotiation and link
connectivity, if not, you will see basic settings for the interface.

17
Network Tools are used to obtain information about network traffic, configuration,
routing and connections.

Tcpdump is a packet capture tool that lists the packets passing through a network
interface. It is useful for checking that traffic is being processed and is therefore
useful for troubleshooting network traffic issues. An example of when this tool would
be used is when a user is not able to make Voice over IP (VoIP) calls to an external
network.

The CLI command is tcpdump to perform this on all hosts. The advanced shell
command is tcpdump –n host <IP address> this command will perform
the tcpdump on the specified host IP address rather than all hosts.

The command will monitor the packet flow into the interface, packet response,
packet drop and ARP information. It can be used to troubleshoot issue such as when
connections cannot be established between servers and clients, traffic is not reaching
another branch office and when traffic is not moving in a VPN tunnel.

Additional options can be added to the command if required like tcpdump llh
which can be used to print the link-level head on each dump line.

Handy Tip:

18
• To stop a command in CLI use CTRL +C this will stop the command from running.

18
Tcpdump pcap is the file format that the result of tcpdump is saved as. This allows
you to view the output of the tcpdump command and analyse it. This is usually done
using a packet analyser tool such as WireShark.

The advanced shell command is tcpdump followed by the options you

require. In this example we have used ‘-i any port 80 -
b’. You will need to then add –w to write the file,
followed by a directory to save the file, in this case we
have chosen to save the file in /tmp/tcpdump.pcap. If we take
the example of a user not being able to make VoIP calls to an external network, the
pcap file allows you to check all transactions between the client phone and external
servers.

Once you have written the pcap file you will need to export the file to read it. To
export the file you would use the command ftpput –u <username> -p
<password> host IP <remote file name> <local file name>
Usually, we recommend that you save the file and then use a packet analyser such as
WireShark to analyse the file.

If you have a customer who has completed the tcpdump and has put the files on their
server for you to analyse you can use the command ftpget –u <username> -
p <password> host ip <local file name> <remote file name>

19
to download the pcap file from the customers server.

Note. You must have an FTP server to download and upload files from Sophos XG
Firewall.

19
Netstat is used to verify all processes and their listening ports.

The advanced shell command to use is netstat –natup. This can be used to
troubleshoot connection issues as it shows the state of a specific service and it’s port
number. For example if a user wanted to find whether a connection to the requested
port is established for SPX service.

20
Connection Status is used to check the total number of connection made via the
Sophos XG Firewall.

The CLI Command to use is system diagnostics utilities

connections count. The CLI Command system diagnostics
utilities connections v4 show src_ip x.x.x.x is used to verify
the stateful table for a specific connection. The output displays the protocol, status of
connection, source/destination IP and port details.

These commands are helpful for troubleshooting firewall rule related issues. An
example of this is if a user wants to verify firewall rule ID, Web filter policy ID, Web
category ID, source and destination IP for specific connection regarding a web
filtering issue.

21
The Routing Table command is used to show and verify the current routing table.

The CLI command is system diagnostics utilities route

runconfig-show. The output includes local, dynamic, static and VPN routes. This
can be used to troubleshoot routing connectivity issues between local and remote
offices.

The advanced shell command is route –n which will return the same information
as the CLI Command. The routing table information is useful in situations like when a
user is trying to reach a remote office and is not able to do so. You want to verify if a
static route exists in the routing table for that connection.

22
Address Resolution Protocol (ARP) maps IP network addresses to the hardware
addresses used by the data link protocol. The protocol operates below the network
layer as part of the interface between the OSI network and OSI link layer.

The CLI command is system diagnostics utilities arp show and is

used to show ARP entries learning from peer device. It can be used to troubleshoot
layer 2 connectivity issues.

For example if a gateway link is down. As part of the troubleshooting steps the user
should check the link status and ARP entries on the device. They would use this
command to check if the XG Firewall is learning the MAC address of the gateway
device. If it is not learning the MAC address of the gateway device they would use the
CLI command system diagnostics utilities arp ping interface
<interface label> <IP Address> to see if it can reach the gateway
device.

23
System Tools allow you to obtain information about the system configuration such as
available space.

The advanced shell command df –h is used to display information about the total
and available space of partitions. The file system parameter specifies the name of the
device on which the file system resides, the directory on which the file system is
mounted, or the relative path name of a file system.

For example if an Appliance has stopped generating reports. One possible cause can
be because the VAR disk partition has used beyond 90% of available space. This
command would be used to verify the total space used in the VAR partition.

24
Free is used to show the total and available memory in the Sophos XG Firewall.

The advanced shell command is free. It displays information about logical RAM
(SWAP).

For example if there are performance issues being reported such as slow browsing of
slow user interface issues, the command can be used to verify the available memory.
It could be that the XG Firewall does not have enough available memory to serve the
requests from the users.

25
The PS command shows the currently running processes.

The advanced shell command is ps | grep <process name> or ps. This

command displays all processes and the path from where the command is being
executed.

The output displays the process ID (pid=PID), Parent Process ID (PPID), Username,
Virtual memory usage of entire process (VSZ), resident set size, the non-swapped
physical memory that a task has used (RSS), multi-character process state (STAT) and
command with all its arguments (CMDLINE). The output is unsorted by default.

This command would be used for example when a user wants to check the PID for
the sandstorm service in order to turn on debug mode for sandboxed service using
KILL command.

26
SF Loader is a customized tool which allows the administrator to perform a number of
actions like resetting the Console password or performing a memory test.

For example if a user has reported slow user interface access. The administrator can
perform an HDD test to look for bad sectors or perform a memory test. If it was an
interface failure the administrator can perform an Ethernet card test to verify any
hardware errors.

Note. Running the SF Loader utility requires downtime for the XG Firewall.

27
The advanced shell command top is a performance tool that provides a dynamic
real-time view of currently running processes, used memory, swap usage and CPU
usage. It sorts tasks by CPU usage, memory usage and run time.

The output displays the CPU load average which allows you to see how much CPU
your computer has been using over periods of time. These periods of time are split
into three, the first is the average over one minute, then the last five minutes and
lastly the last fifteen minutes. To give you an idea of these numbers, a completely
idle computer has a load average of 0.

This command is usually used for troubleshooting scenarios like Sophos XG Firewall
performance issues such as high CPU, high memory and slow internet browsing.

28
nslookup is a DNS tool that can be used to troubleshoot scenarios where a website is
unable to be opened or if a user is unable to reach a domain using a specific gateway.

The CLI command is dnslookup host <domain name>.

The advanced shell command is nslookup <domain name>.

These commands are used to resolve a domain name such as www.sophos.com to an

IP address. It is also used to resolve an IP address to a domain name.

29
IPset is used to inspect IP sets using the Advanced Shell.

An IP set will store IP addresses, (TCP/UDP) port numbers, IP and MAC address pairs,
IP address and port number pairs.

The ipset command can be used to return details about a specific set or provide a list
of all IPset tables.

The commands to be used are

• ipset –L which will output a list of al IPset tables.
• ipset -L <set name> which will output the details of a specific set.

30
Sophos XG Firewall tracks the state of connections between hosts. A connection state
table is maintained for all traffic that passes through the Sophos XG Firewall. Any
communication that does not match an existing connection, or does not match rules
allowed for new connections, is dropped.

The connection table can be viewed by either going to the WebAdmin console
MONITOR & ANALYZE > Diagnostics > Connection List or through the Advanced Shell
Command via CLI.

Using the command conntrack you can view and manage the connection tracking
state table. The advanced shell command is conntrack -L | grep <ip
address> which displays the stateful inspection table of the Sophos XG Firewall. It
displays the protocol name, number, source IP, destination IP, source port, destination
port, connection status, mac address details, firewall rule ID and policy detail. The
mark value e.g. 0x8003 shows the gateway which was used which can help identify
issues.

31
Connection Tracking helpers are modules that are able to assist in tracking protocols
like FTP, SIP and H.323 which use separate control and data connections for data
transfers. The helpers create expectations which are stored in a separate table.
Expectations are used to determine if a connection with specific parameters is related
to the previous connection.

To demonstrate how this works we will look at how FTP connects.

FTP uses separate control and data connections between the client and the server.
Once the FTP connection is established;

• The Control Connection is established on port 21 of the FTP server. It sends out a
connection command to the client
• The client responds to the connection command from the server on the same port
(21)
• The FTP server responds to the client with the data connection on port 20
• The client will then confirm the connection and will be able to use the connection
for secure data transfer

The tracking helpers would determine that a connection on port 20 will follow a
connection on port 21 and allow the connections for successful data transfer.

32
If the connection is not successful, you can use the following commands to confirm
the connection status of the protocol you are seeing issues with.

You can use the advanced shell command conntrack –E which returns all live
connections along with their current status.

conntrack –E expect returns details about the expected connection based on

the existing connection.

• system system_modules show returns a list of the protocols that are

currently loaded (able to be used).
• system system_modules <protocol> unload will unload the
protocol specified.
• system system_modules <protocol> load will load the protocol
specified.

33
When traffic is routed through an interface, the Sophos XG Firewall first checks the
advanced firewall settings. If the traffic is valid according to the defined settings then
it is routed to it’s intended destination.

To check the advanced firewall settings, access CLI and navigate to the CLI menu, run
the command show advanced-firewall

You will notice that the output of the command shows that Strict Policy is on. This is
always on by default and is used to prevent the following legacy attacks;
• Winnuke attack . In a Winnuke attack, the attacker sets the rarely used URG flag
(Urgent flag) in the TCP header, which tells the receiving system to process the
packet urgently. Within this time, it creates a loop and the receiving system goes
down due to high utilization of CPU.
• Land attack . Known as LAN Denial attack, is a DoS attack targeted on a system
within a LAN. It sends a spoofed packet to the receiver which in turn locks it up.
• TCP/UDP destination/source port 0. These type of attacks are targeted to make the
remote system go high on CPU utilization. In these types of attacks, TCP or UDP
destination/source port is 0. Since this port cannot be reached, the remote system
keeps on retrying and ends up in a loop.
• Data on TCP Sync. Many times attackers send data on the TCP Sync Packet. This
type of data is usually system commands or spoofed messages which are able to
lock a remote computer.

34
• Zero IP Protocol. When a zero is set in the sender IP, the receiver keeps on trying to
send an ACK to 0.0.0.0, however since this address is not valid, the receiver system
goes down.
• TTL Value 0. When TTL value is set to 0, the receiver cannot send any messages to
the sender system. For every communication, it receives an error “transmit failed”.
This type of attack makes the receiver system inoperative of any communication
because the immediate router will have TTL 0, which means all communication to
the immediate router will be dropped.

By Default, Strict Policy is always on. To turn the strict policy on or off go to use the
CLI command set advanced-firewall strict-policy on/off.

34
You can deploy High Availability (HA) in one of two deployment modes; Active-Active
and Active-Passive.

Active-Active mode can be enabled using a single license on one device, however,
after failover the network will no longer be protected as no license is activated on the
auxiliary device. It is recommended to have an individual licence on both devices.
License subscription should be the same on both the devices so that after failover,
the policies will work as expected.

Active-Passive mode requires only a single license on the Active device. At the time of
failover, the auxiliary device takes the license subscription and starts processing
packets.

35
When HA is enabled, both devices will start the process to form the HA cluster. This
process is completed in 3 steps by the primary device;

Step 1 is the Sanity Check. The CSC service checks whether HA can be enabled on
both devices. It checks the model number, vendor details and firmware version on
both the devices and confirms that the HA status is disabled on the auxiliary device.
Additionally, it confirms that there is no alias or VLAN configured on the dedicated HA
port and that no override MAC addresses exist on the dedicated HA port. It then
checks that the Speed/duplex and MTU/MSS are set to the default settings on both
the devices. It confirms that SSH is enabled on the DMZ zone for the dedicated port.
This is so that the primary device can connect on port 22 and push the configuration
file. Lastly it confirms that all monitoring ports are in an UP state. Once completed, it
connects to the auxiliary device on port 22 using the passphrase configured.

Step 2 is to prepare the system for HA. The primary device synchronizes the
configuration and connection tracking files with the auxiliary device. The primary
device generates virtual MAC addresses and synchronizes them with the auxiliary
device and synchronizes the original MAC addresses on both the devices. It then
generates the configuration file for HA.

Step 3 is to synchronize the configuration. The msync service will perform a soft boot
on the auxiliary device. Once the auxiliary device comes up it will join the HA cluster.

36
During the time when the auxiliary device is joining the cluster it will synchronize with
the primary device configuration file. The primary device will remain in a frozen state
until the configuration is complete.

36
HA logs can be monitored via the CLI. The HA status can be checked by using the CLI
command system ha show details. To check HA logs via the advanced shell
use the command tail -f /log/applog.log | grep ha

The logs can be checked to ensure that HA has been configured successfully and to
check for any errors if there are issues when enabling HA;
• fwm:enableha successfully done means that the sanity check has
completed on the local appliance
• enableha on peer done confirms that HA is enabled on the peer appliance
• enableha: HA is enabled now confirms that the sanity check has
completed on the peer appliance

Once the sanity check is performed on the appliances, the primary appliance will
change it’s state to 5:2 meaning it will become a standalone device first before
primary-auxiliary state.

37
A common issue you may see is when the administrator is unable to activate HA.

The administrator reports that they have followed the configuration steps however,
when they complete the configuration the following error is returned in the user
interface, ‘Unable to connect with peer device’.

38
The first thing to do to determine the root cause is to check the knowledgebase for
the error returned. In this case, we will assume that no articles are available for this
error.

The next step would be to check the log files for any errors. The applog log file on the
advanced shell can be checked using the command tail –f
/log/applog.log

In this example the following error is returned in the logs:

Jan 31 02:11:54 enableha: enableha called from GUI
Jan 31 02:12:09 enableha: peer sanity check failed ! ! !

This error indicates that the appliance has not met the HA prerequisites or that the
device has not been configured with the correct passphrase.

39
The next step is to perform verification testing on the device. To do this, verify the
passphrase used in HA configuration by re-entering the passphrase. Select Enable HA
and confirm if you see the confirmation message that HA is now enabled.

It is recommended that you check connectivity to the peer administration IP and

dedicated port IP and ensure that SSH access is enabled for the DMZ zone on both
appliances.

Other possible causes for this error to be returned can be due to the appliances not
being the same model or having the same firmware version. Alternatively, if the WAN
link has DHCP or PPPoE configured or an alias or VLAN interface has been configured
on the dedicated HA port.

40
Complete the following tasks in Lab 1
Task 1: Register and Activate Sophos XG Firewall

41
Please take a few minutes to answer the following knowledge check questions.

If you find that you are unable to answer the questions in this knowledge check
section, please take some time to review the material covered in this course.

42
43
44
45
46
47
48
On completion of this Module, you can now:
 Understand the troubleshooting process
 Understand how to access Sophos XG Firewall to perform troubleshooting
 Identify key troubleshooting tools
 Understand how each tool is used and when it should be used
 Understand High Availability
 Troubleshoot High Availability issues

49
Now that you have completed this module, you should complete Module 802:
Network Protection.

50
51