FW4020 20.0v1 Troubleshooting Web Scanning On Sophos Firewall

Copyright © 2024 Sophos Ltd
Troubleshooting Web
Scanning on Sophos
Firewall
Sophos Firewall
Version: 20.0v1
[Additional Information]
Sophos Firewall
FW4020: Troubleshooting Web Scanning on Sophos Firewall
January 2024
Version: 20.0v1
© 2024 Sophos Limited. All rights reserved. No part of this document may be used or reproduced in any form or by any means without the prior written
consent of Sophos.
Sophos and the Sophos logo are registered trademarks of Sophos Limited. Other names, logos and marks mentioned in this document may be the
trademarks or registered trademarks of Sophos Limited or their respective owners.
While reasonable care has been taken in the preparation of this document, Sophos makes no warranties, conditions or representations (whether express
or implied) as to its completeness or accuracy. This document is subject to change at any time without notice.
Sophos Limited is a company registered in England number 2096520, whose registered office is at The Pentagon, Abingdon Science Park, Abingdon,
Oxfordshire, OX14 3YP.
Troubleshooting Web Scanning on Sophos Firewall - 1

Copyright
Copyright ©
© 2024
2023 Sophos
Sophos Ltd
Ltd
Troubleshooting Web Scanning on Sophos Firewall

In this chapter you will learn the RECOMMENDED KNOWLEDGE AND EXPERIENCE
logs and commands that can be ✓ How to configure web protection on Sophos
used to troubleshoot web Firewall
scanning when using the DPI
engine or using a web proxy.
DURATION 6 minutes
In this chapter you will learn the logs and commands that can be used to troubleshoot web scanning
when using the DPI engine or using a web proxy.

Logging for DPI Scanning
If web protection is configured to use DPI scanning, the log viewer will provide useful information for
troubleshooting, and this should always be the place to start.
If you are not able to determine what is happening to resolve the issue from the log viewer, you may
need to enable web debug logging for the DPI engine.

Additional information in
the notes
Optionally customize the debug mask
Component Hex Code Mask

DEBUGP_CONFIG 0x01 1
DEBUGP_ACL 0x02 2
DEBUGP_WEBCAT 0x10 16
DEBUGP_AV 0x20 32
DEBUGP_POLICY 0x100 256
DEBUGP_SSL 0x8000 32768
It is debugp not just
DEBUGP_QUOTA 0x20000 131072
debug!
echo mask=$((0x10+0x20)) > /var/tmp/debugp.conf
Toggle web filtering debug logging on/off service ips:debugp -ds nosync
Review /log/ips.log tail -f /log/ips.log
The DPI engine relies on Snort and the IPS service, and so it is in the IPS service that debug logging is
enabled for web scanning. When you enable debug logging, it can log such a vast volume of data that
it makes it hard to see what is going on. Before enabling the debug logging, you can optionally create a
mask that will enable debugging for only specific components, allowing you to focus your
troubleshooting.
To create a mask, you echo the values for the components into a configuration file using the Advanced
Shell. You can see an example here for web categorization and antivirus. You then need to toggle on
the debug logging. Please note that this uses ‘debugp’ instead of plain ‘debug’. This will then log to the
file /log/ips.log.
[Additional Information]
Full set of debugp masks
DEBUGP_NONE 0x00 // 0
DEBUGP_CONFIG 0x01 // 1
DEBUGP_ACL 0x02 // 2
DEBUGP_EPOLL 0x04 // 4
DEBUGP_WEBCAT_INIT 0x08 // 8
DEBUGP_WEBCAT 0x10 // 16
DEBUGP_AV 0x20 // 32
DEBUGP_AV_CACHE 0x40 // 62
DEBUGP_LOG 0x80 // 128
DEBUGP_POLICY 0x100 // 256
DEBUGP_REQ_FSM 0x200 // 512
DEBUGP_DECOMPRESS 0x400 // 1024

DEBUGP_MIME 0x800 // 2048
DEBUGP_POOL 0x1000 // 4096
DEBUGP_HOLD_FSM 0x2000 // 8192
DEBUGP_TCP_HOLD 0x4000 // 16384
DEBUGP_SSL 0x8000 // 32768
DEBUGP_NSE_CACHE 0x10000 // 65536
DEBUGP_QUOTA 0x20000 // 131072
DEBUGP_UTIL 0x40000 // 262144
DEBUGP_MICRO (DEBUGP_EPOLL | DEBUGP_WEBCAT | DEBUGP_AV
| DEBUGP_AV_CACHE | DEBUGP_LOG)
DEBUGP_DEFAULT (DEBUGP_CONFIG | DEBUGP_POLICY |
DEBUGP_REQ_FSM | DEBUGP_MICRO | DEBUGP_SSL |
DEBUGP_TCP_HOLD)


SFVUNL_HV01_SFOS 18.0.1 MR-1-Build396# tail -f /log/ips.log
[Sep 24 08:55:32 :5523]:signo_handler: got signal 512
[Sep 24 08:55:32 :5523]:setVariable: set signal 54
1600955732.129293900 [10565/0x0] [nsg_web_config_reload.c:88:process_sig_event] [5523] signal: Real-time signal 20
1600955732.129322200 [10565/0x0] [nsg_debugp/nsg_debugp.c:144:parsed_line_handler] debugp.conf: section=, name=mask, value=16
1600955732.129319600 [10561/0x0] [nsg_web_config_reload.c:88:process_sig_event] [5523] signal: Real-time signal 20
1600955732.129330900 [10565/0x0] [nsg_debugp/nsg_debugp.c:71:nsg_debugp_cfg_set_nsg_debugp_mask] Parsed [mask] -> [16=0x10]
1600955732.129345600 [10565/0x0] [nsg_debugp/nsg_debugp.c:299:nsg_debugp_toggle] Setting debug mask set to 0x10
1600955732.129358200 [10561/0x0] [nsg_debugp/nsg_debugp.c:144:parsed_line_handler] debugp.conf: section=, name=mask, value=16
1600955732.129430400 [10561/0x0] [nsg_debugp/nsg_debugp.c:71:nsg_debugp_cfg_set_nsg_debugp_mask] Parsed [mask] -> [16=0x10]
1600955732.129443200 [10561/0x0] [nsg_debugp/nsg_debugp.c:299:nsg_debugp_toggle] Setting debug mask set to 0x10
1600955734.685482200 [10549/0xbe5f0000014f] [nsg_nse_policy.c:1638:do_sni_webcat] ===> Submit to urlcat
1600955734.685957400 [10566/0x0] [nsg_webcat.c:271:webcat_url_ingress_queue_drain] Send to webcat_server-> type: 4 index: 1 url:
client.wns.windows.com
1600955734.806726100 [10549/0xd45b000000be] [nsg_nse_policy.c:1638:do_sni_webcat] ===> Submit to urlcat
1600955741.896876600 [10548/0xbe6900000148] [nsg_nse_policy.c:1638:do_sni_webcat] ===> Submit to urlcat
1600955742.011769500 [10549/0xd457000000ba] [nsg_nse_policy.c:1638:do_sni_webcat] ===> Submit to urlcat
1600955743.414159700 [10549/0xd455000000bc] [nsg_state_req_new_request.c:251:event_fcn_url_parsed] ===> Submit to urlcat
sophostest.com/gambling/index.html
1600955743.521190700 [10566/0xd455000000bc] [nsg_request_fsm.c:690:request_fsm_urlcat_response] For url
sophostest.com/gambling/index.html the category ids
1600955743.521214200 [10566/0xd455000000bc] [nsg_request_fsm.c:692:request_fsm_urlcat_response] ids[0] = 21
1600955743.521218800 [10566/0xd455000000bc] [nsg_request_fsm.c:710:request_fsm_urlcat_response] ===> Got urlcat result
Here you can see an example of the ips.log with debugging enabled for just web categorization.
First, you can see the debug mask being set.
Further down you can then see the request for categorization and the result.

Slow Browsing Through Web Proxy 1
We’ll now look at a scenario where Sophos Firewall is configured to use the web proxy instead of DPI
web scanning, and users are reporting slow browsing.


With proxy
SF01V_HV01_SFOS 19.0.0 GA-Build317# curl -x http://172.16.16.16:3128 -L

http://ipv4.download.thinkbroadband.com/100MB.zip > /dev/nul
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 100M 100 100M 0 0 13.4M 0 0:00:07 0:00:07 --:--:-- 16.8M
Without proxy
SF01V_HV01_SFOS 19.0.0 GA-Build317# curl -L

100 100M 100 100M 0 0 36.7M 0 0:00:02 0:00:02 --:--:-- 36.7M
When using the web proxy, it is expected that there is a reduction in throughput of up to 25%.
When troubleshooting reports of slow browsing, start by comparing the throughput with and without
the web proxy. The Advanced Shell can be used for testing. The first command configures proxy
settings before using curl to download a .zip file. The second command downloads the file without a
proxy.
In this example, you can see that, with the proxy, the throughput is approximately 30% of the
throughput without the proxy, a reduction of 70%. This may indicate an issue that needs to be
resolved.

SFVUNL_HV01_SFOS 18.0.1 MR-1-Build396# service awarrenhttp:debug –ds nosync

200 OK
SFVUNL_HV01_SFOS 18.0.1 MR-1-Build396# tail –f /log/awarrenhttp_access.log
1600943034.755351000 [ 6948/0x7f273defd800] fwid=2 fwflag="VS" iap=12 aap=9
conn_id=3903150656 id="0001" name="http access" action="pass" method="GET"
srcip="172.16.16.10" dstip="13.227.219.100" user="administrator@sophos.local" statuscode=304
cached=0 trxlen=599 rxlen=382 url="http://sophostest.com/" referer="" type=""
upload_file_name="" upload_file_type="" download_file_name="" download_file_type=""
authtime=0 dnstime=1573 cattime=100797 avscantime=0 fullreqtime=115930 ua="Mozilla/5.0
(Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.102
Safari/537.36" activity="" av_transaction_id="" categoryname="Information Technology"
category="29" app_id=0 app_name="None" app_cat="None" exceptions=""
To get more detailed information, enable debug logging for awarrenhttp – this is the web proxy.
Transactions will be logged to /log/awarrenhttp_access.log.
For each transaction you can see how long different parts of the transaction took. These are measured
in microseconds.

Action Description Expected Max Duration (microseconds)

authtime Time for authenticating the request Up to 5,000,000
dnstime Time for resolving the DNS Up to 50,000
cattime Time for identifying the category of the site Up to 1,000,000
avscantime Time for malware scanning Can depend on size of content
fullreqtime Total time to process the request Can depend on size of content
The five times recorded for transactions are:

• authtime, the time for authenticating the request. This should not exceed 5 million microseconds.
• dnstime, the time for DNS resolution. This should not exceed 50 thousand microseconds.
• cattime, the time for identifying the category of the site. This should not exceed 1 million
microseconds.
• avscantime, the time for malware scanning. This can depend on the size of the content.
• fullreqtime, the total time to process the request. This can depend on the size of the content.

High authtime?
▪ Check the authentication configuration

▪ Check for connectivity and responsiveness to authentication server(s)
▪ Check resource utilization on authentication server(s)
If you are seeing a high authentication time, check the configuration for authentication servers.
For each authentication server, check the connectivity and responsiveness and ensure that the most
responsive is at the top of the list.
Check the resource utilization on the authentication servers. If they are overloaded this could cause
delays.

High dnstime?
▪ Check DNS resolution manually
SFVUNL_HV01_SFOS 18.0.1 MR-1-Build396# host -a sophostest.com 172.16.16.16

Trying "sophostest.com"
;; Warning: Message parser reports malformed message packet.
Trying "sophostest.com"
Using domain server:
Name: 172.16.16.16
Address: 172.16.16.16#53
Aliases:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 43098

;; flags: qr rd ra; QUERY: 1, ANSWER: 8, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;sophostest.com. IN ANY
;; ANSWER SECTION:
sophostest.com. 21599 IN NS ns-93.awsdns-11.com.
Received 224 bytes from 172.16.16.16#53 in 1 ms
If you are seeing a high DNS time, check the time to resolve various domains using the Sophos
Firewall. Try to include domains that are unlikely to be in the cache to get the most real-world figures.
You can do this from the advanced shell with the command: host -a <domain> <Sophos
Firewall IP address>
Please note that for this example, the output has been truncated.

SF01V_HV01_SFOS 19.0.0 GA-Build317# curl -x http://172.16.16.16:3128 -L

100 100M 100 100M 0 0 31.19M 0 0:00:03 0:00:03 --:--:-- 31.19M
SF01V_HV01_SFOS 19.0.0 GA-Build317# curl -L

100 100M 100 100M 0 0 36.7M 0 0:00:02 0:00:02 --:--:-- 36.7M
Once you have resolved the cause of any high time values, re-test with and without a proxy.
The time with the proxy should be within 25% of the time without the proxy.

Chapter Review
Debug logging for DPI can be enabled using the command ips:debugp –ds nosync which will then log to
the file /log/ips.log. Optionally a mask can be created to enable debugging only for specific components.
The Advanced Shell can be used to test throughput for web requests. The curl command can be used to
download a file, with and without a proxy configured. The reduction in throughput with a proxy is
expected to be up to 25%.
Debug logging can be enabled for awarrenhttp, which is the web proxy. This will show times recorded for
transactions, including authenticating the request (authtime) and the time for DNS resolution (dnstime).
Here are the main things you learned in this chapter.
Debug logging for DPI can be enabled using the command ips:debugp -ds nosync which will then log
to the file /log/ips.log. Optionally a mask can be created to enable debugging only for specific
components.
The Advanced Shell can be used to test throughput for web requests. The curl command can be used
to download a file, with and without a proxy configured. The reduction in throughput with a proxy is
expected to be up to 25%.
Debug logging can be enabled for awarrenhttp, which is the web proxy. This will show times recorded
for transactions, including authenticating the request (authtime) and the time for DNS resolution
(dnstime).


FW4020 20.0v1 Troubleshooting Web Scanning On Sophos Firewall

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

FW4020 20.0v1 Troubleshooting Web Scanning On Sophos Firewall

Uploaded by

Copyright:

Available Formats

Copyright © 2024 Sophos Ltd

Troubleshooting Web Scanning on Sophos Firewall - 1

Troubleshooting Web Scanning on Sophos Firewall

Troubleshooting Web Scanning on Sophos Firewall - 2

Logging for DPI Scanning

Troubleshooting Web Scanning on Sophos Firewall - 3

Component Hex Code Mask

echo mask=$((0x10+0x20)) > /var/tmp/debugp.conf

Review /log/ips.log tail -f /log/ips.log

Troubleshooting Web Scanning on Sophos Firewall - 4

Troubleshooting Web Scanning on Sophos Firewall - 4

Logging for DPI Scanning

First, you can see the debug mask being set.

Troubleshooting Web Scanning on Sophos Firewall - 5

Slow Browsing Through Web Proxy 1

Troubleshooting Web Scanning on Sophos Firewall - 6

Slow Browsing Through Web Proxy 2

SF01V_HV01_SFOS 19.0.0 GA-Build317# curl -x http://172.16.16.16:3128 -L

SF01V_HV01_SFOS 19.0.0 GA-Build317# curl -L

Troubleshooting Web Scanning on Sophos Firewall - 7

Slow Browsing Through Web Proxy 2

SFVUNL_HV01_SFOS 18.0.1 MR-1-Build396# service awarrenhttp:debug –ds nosync

Transactions will be logged to /log/awarrenhttp_access.log.

Troubleshooting Web Scanning on Sophos Firewall - 8

Slow Browsing Through Web Proxy 2

Action Description Expected Max Duration (microseconds)

The five times recorded for transactions are:

Troubleshooting Web Scanning on Sophos Firewall - 9

Slow Browsing Through Web Proxy 2

▪ Check the authentication configuration

Troubleshooting Web Scanning on Sophos Firewall - 10

Slow Browsing Through Web Proxy 2

▪ Check DNS resolution manually

SFVUNL_HV01_SFOS 18.0.1 MR-1-Build396# host -a sophostest.com 172.16.16.16

;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 43098

Received 224 bytes from 172.16.16.16#53 in 1 ms

Troubleshooting Web Scanning on Sophos Firewall - 11

Slow Browsing Through Web Proxy 3

SF01V_HV01_SFOS 19.0.0 GA-Build317# curl -x http://172.16.16.16:3128 -L

SF01V_HV01_SFOS 19.0.0 GA-Build317# curl -L

Troubleshooting Web Scanning on Sophos Firewall - 12

Here are the main things you learned in this chapter.

Troubleshooting Web Scanning on Sophos Firewall - 18

Troubleshooting Web Scanning on Sophos Firewall - 19

You might also like