Forcepoint User ID

troubleshooting
Introduction
This document provides tips and instructions how to troubleshoot Forcepoint User ID related issues.

Troubleshooting Forcepoint User ID operation

Like with any troubleshooting it is best to use systematic approach. With Forcepoint User ID (FUID) you
should start troubleshooting from Next Generation Firewall side. There you can first check if name cache
shows any user information:
# cat /proc/stonegate/name-cache/names

root@fw-ngfw-1402c1-1:~# cat /proc/stonegate/name_cache/names |grep -i 0x001

V:0 Administrator@<DOMAIN> 0x001 -999 -2 1 <IP ADDRESS OF HOST>
V:0 jdoe@<DOMAIN> 0x001 -999 -2 1 <IP ADDRESS OF HOST>
V:0 Administrator@<DOMAIN> 0x001 -999 -2 1 <IP ADDRESS OF HOST>

If user and group information is seen, that means that FUID is working, or at least it was working as at some
point as engine has received data from FUID.

User shown in name cache but newly added group missing

If user is found from the list but some newly added AD groups, that user belongs to, aren’t shown, run
following command on FUID server to tell DAS to manually fetch the updated user and group information
from AD by opening connection to global catalog port 3268:
# touch /opt/Forcepoint/bin/das.ini

This works also with newly added users as even if FUID has seen the logon event in DC security logs, it
won’t include the user unless DAS has also read the user information from the LDAP tree.

No user information in the NGFW name cache

If there’s no user information shown in the NGFW name cache, check first that configuration looks fine:
# cat /data/config/policy/latest/fuid.sgc

root@fw-ngfw-1402c1-1:~# cat /data/config/policy/latest/fuid.sgc

# StoneGate Management Center 6.3.3 [10432] <http://www.stonegate.com>
# Copyright 2000-2017 Forcepoint. All rights reserved. <https://www.forcepoint.com/product/network-
firewall/forcepoint-stonesoft-next-generation-firewall>

fuid {
ip_address <IP ADDRESS OF FUID SERVER>;
port 5000;
tls_profile 8;
server_timeout 10;
cache_expiration 300;
tls_server_identity {
cname "fuid.<DOMAIN>";
};

2
Verify that ip_address shows the FUID Linux server address, port is 5000, and tls_server_identity
matches the certificate. You can verify certificate CN on the FUID server CLI with following command:
# openssl x509 -in /opt/Forcepoint/bin/UID_Server/UserIDServer.crt -text |grep -i subject:

[root@fuid ~]# openssl x509 -in /opt/Forcepoint/bin/UID_Server/UserIDServer.crt -text |grep -i subject:

Subject: CN=fuid.<DOMAIN>

If configuration looks good, check if NGFW has been able to establish connection with FUID:
# netstat -pan |grep 5000

root@fw-ngfw-1402c1-1:~# netstat -pan |grep 5000 |grep fuid

tcp 0 0 <IP ADDRESS OF NGFW>:8564 <IP ADDRESS OF FUID SERVER>:5000
ESTABLISHED 4079/fuid

No connection established with FUID

If connection hasn’t been established, you’ll need to do the basic network troubleshooting. Test first the
basic TCP 5000 connectivity with telnet:
# telnet <IP ADDRESS OF FUID SERVER> 5000

root@fw-ngfw-1402c1-1:~# telnet <IP ADDRESS OF FUID SERVER> 5000

Trying <IP ADDRESS OF FUID SERVER>...
Connected to <IP ADDRESS OF FUID SERVER>.
Escape character is '^]'.

If basic TCP handshake works, you’ll know that there’s at least some level of connectivity. You can then
collect traffic captures from both the NGFW and FUID server while restarting FUID daemon on the NGFW
with msvc -d fuid and msvc -u fuid command, and then compare the captures to see if there’s e.g. path
MTU discovery issues in the network.

TIP: If tcpdump binaries haven’t been installed on CentOS 7 server, use yum -y install tcpdump command
to install it (requires FUID server to have Internet access).

If captures do not show any issues, it is best to open support case so tech support team can check the
traces from both NGFW and FUID side.

If the manual connection attempt fails, check with tcpdump if you see any reply packets from FUID server,
and if SYN packets are reaching FUID server. On the FUID server verify that it is listening to port 5000:
# netstat -pan |grep 5000

Tip: netstat command will work only if net-tools package has been installed. To install this package, use
yum -y install net-tools command.

If you see FUID listening on port 5000, but no packets are seen from NGFW, then you’ll know that issue is
in the network between NGFW engine and FUID server. If tcpdump on FUID server shows SYN packets
coming in, but FUID server never replies to those and you also don’t see the connection attempts with
netstat, then most likely Linux software firewall is blocking the connections, and you’ll need to add the rules.
On CentOS 7 you can use the commands shown at page 4.

If FUID connection is established, and with tcpdump you don’t see any issues, then problem is likely on
FUID side.

3
Troubleshooting FUID service operation
On FUID server check first if UID service has received any information about users with IP associated with
them from DC Agent:
# curl -i -k /opt/Forcepoint/bin/UID_Server/UserIDServer.crt -H "Accept: application/json"
https://127.0.0.1:5000/api/uid/v1.0/users?ip_only=true

NOTE! Depending on where and how FUID was installed, certificate name and path might be different.
Above is the name of certificate created internally during FUID installation and path where it is located if
default installation path was used.

[root@fuid ~]# curl -i -k /opt/Forcepoint/bin/UID_Server/UserIDServer.crt -H "Accept: application/json"

https://127.0.0.1:5000/api/uid/v1.0/users?ip_only=true
curl: (3) <url> malformed
HTTP/1.1 200 OK
Content-Type: application/json
Last-Modified: 1522686149.669841
Content-Length: 1000

{
"users": [
{
"NTLMIdentity": "AD\\Administrator",
"changetype": "ip-add",
"dn": "CN=Administrator,CN=Users,DC=<DOMAIN>,DC=<DOMAIN>,DC=<DOMAIN>",
"groups": [
"CN=Domain Admins,CN=Users,DC=<DOMAIN>,DC=<DOMAIN>,DC=<DOMAIN>",
"CN=Schema Admins,CN=Users,DC=<DOMAIN>,DC=<DOMAIN>,DC=<DOMAIN>",
"CN=Group Policy Creator Owners,CN=Users,DC=<DOMAIN>,DC=<DOMAIN>,DC=<DOMAIN>",
"CN=Domain Users,CN=Users,DC=<DOMAIN>,DC=<DOMAIN>,DC=<DOMAIN>",
"CN=Administrators,CN=Builtin,DC=<DOMAIN>,DC=<DOMAIN>,DC=<DOMAIN>",
"CN=Enterprise Admins,CN=Users,DC=<DOMAIN>,DC=<DOMAIN>,DC=<DOMAIN>"
***CLIP***

In the case where FUID user list shows details, but NGFW name cache does not, you'll need to go back
investigating the communication between NGFW and FUID as user info isn't reaching NGFW. Trace files
might also need to be investigated. On the NGFW side FUID daemon traces are written to
/data/diagnostics/fuid-yyyymmdd-hhmmss-xxxx files, and main trace file for FUID service on FUID
server side is /opt/Forcepoint/bin/UID_Server/uid_service.log. However the message syntax on these
files isn’t publicly documented so you should open support case so tech support can help with investigating
the traces and logs on NGFW and FUID server.

4
Troubleshooting DAS service operation
In case FUID doesn’t show any users, check if DAS is able to read the LDAP database to get user and
group information from AD:
# curl -i -k /opt/Forcepoint/bin/UID_Server/UserIDServer.crt -H "Accept: application/json"
https://127.0.0.1:5000/api/uid/v1.0/users?ip_only=false

[root@fuid ~]# curl -i -k /opt/Forcepoint/bin/UID_Server/UserIDServer.crt -H "Accept: application/json"

https://127.0.0.1:5000/api/uid/v1.0/users?ip_only=false
curl: (3) <url> malformed
HTTP/1.1 200 OK
Content-Type: application/json
Last-Modified: 1522686149.669841
Content-Length: 7891

{
"users": [
{
"NTLMIdentity": "AD\\Guest",
"changetype": "add",
"dn": "CN=Guest,CN=Users,DC=<DOMAIN>,DC=<DOMAIN>,DC=<DOMAIN>",
"groups": [
"CN=Domain Guests,CN=Users,DC=<DOMAIN>,DC=<DOMAIN>,DC=<DOMAIN>",
"CN=Guests,CN=Builtin,DC=<DOMAIN>,DC=<DOMAIN>,DC=<DOMAIN>"
***CLIP***

User and group information should be shown, and that means DAS has been able to get data from AD via
global catalog. If FUID still does not show any user information even when DAS is able to read the LDAP
content, verify that FUID (uwsgi) is listening on port 5001:
# netstat -pan |grep 5001

[root@fuid ~]# netstat -pan |grep 5001

tcp 0 0 0.0.0.0:5001 0.0.0.0:* LISTEN 3741/uwsgi

And make sure also that you added rule for port 5001 to the Linux software firewall running on FUID server.

If DAS hasn’t been able to fetch user and group data from AD global catalog, trigger DAS to query data
from AD while tcpdumping:
# tcpdump -nni <interface_ID> port 3268

NOTE! If global catalog port is different than 3268, adjust filter accordingly. You can of course always
capture to file.

On second SSH/CLI session run following command to manually tell DAS to fetch the information from AD
global catalog:
# touch /opt/Forcepoint/bin/das.ini

You should see DAS opening connection to AD global catalog port, and AD sending data to DAS service.
Note that depending on AD database size, this connection might last for a while. After connection is closed,
check again if DAS has managed to get user and group data from AD:
# curl -i -k /opt/Forcepoint/bin/UID_Server/UserIDServer.crt -H "Accept: application/json"
https://127.0.0.1:5000/api/uid/v1.0/users?ip_only=false

5
If DAS still doesn’t show any user information or specific users are missing, one possibility could be that
users do not have email specified, and by default DAS ignores those users. To disable this check add
NoEmptyEmail=0 to the [DAS] section of /opt/Forcepoint/bin/das.ini.

If connection was very short, and was e.g. closed right after TCP handshake, then likely there's some
communication problems between DAS and AD server that needs to be investigated further with tcpdumps.
You can also check the DAS trace file /opt/Forcepoint/bin/WebsenseDAService.log, but as with FUID
daemon traces on NGFW and FUID service logs, syntax of these logs aren’t publicly available, and thus it is
best to open tech support case.

If no replies are seen from AD on global catalog port, verify that global catalog port hasn’t been changed on
the AD side, and that AD is listening on global catalog port:
> netstat -an |findstr 3268

C:\Users\Administrator>netstat -an |findstr 3268

TCP 0.0.0.0:3268 0.0.0.0:0 LISTENING
UDP 0.0.0.0:53268 *:*

You can also test manually sending LDAP query to global catalog port using curl:
> curl -k -v -u "<AD_admin_account>"
"ldap://<AD_server_address>:3268/<base_DN>?cn,objectClass?sub?(cn=<CN_to_search>)"

# curl -k -v -u "Administrator@<DOMAIN>" "ldap://<IP ADDRESS OF AD

SERVER>:3268/cn=users,DC=<DOMAIN>,DC=<DOMAIN>,DC=<DOMAIN>?cn,objectClass?sub?(cn=*do
e*)"
Enter host password for user 'Administrator@<DOMAIN>':
* About to connect() to <IP ADDRESS OF AD SERVER> port 3268 (#0)
* Trying <IP ADDRESS OF AD SERVER>...
* Connected to <IP ADDRESS OF AD SERVER> (<IP ADDRESS OF AD SERVER>) port 3268 (#0)
* LDAP local: ldap://<IP ADDRESS OF AD
SERVER>:3268/cn=users,DC=<DOMAIN>,DC=<DOMAIN>,DC=<DOMAIN>?cn,objectClass?sub?(cn=*do
e*)
DN: CN=John Doe,CN=Users,DC=<DOMAIN>,DC=<DOMAIN>,DC=<DOMAIN>
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: user

cn: John Doe

* Connection #0 to host <IP ADDRESS OF AD SERVER> left intact

Windows firewall or other software firewall installed on AD server could also block the connections from
DAS, and thus you might need to add rule for these connections.

6
Troubleshooting DC Agent
If DAS shows user and group information but FUID does not show any users with IP associated with them,
while FUID is able to communicate with DC Agent, try first having someone logging into computer in the
domain and see if user gets added to FUID user listing. If it does not, issue is likely that DC Agent isn't
working/installed properly or that DC Agent isn’t able to read the DC security log. One reason for this could
be use of AD credentials that do not have enough permissions. This should be visible in dcagent.log file
that by default is found in C:\Program Files\Forcepoint\DC Agent folder. The syntax of this log file isn’t
publicly documented, which means that you might need to open tech support case.

Other issue could be that DC Agent isn't able to communicate with Domain Controller at all. Traffic could be
captured with Wireshark to see if this is the case, and you can also check netstat output on both DC Agent
server and Domain Controller to see if connections are getting established. On DC Agent Windows server
you can use e.g. this command:
> netstat -an |findstr <DC_IP_address>

C:\Users\Administrator.AD>netstat -an |findstr <IP ADDRESS OF AD SERVER>

TCP <IP ADDRESS OF FUID>:54620 <IP ADDRESS OF AD SERVER>:135 ESTABLISHED
TCP <IP ADDRESS OF FUID>:54621 <IP ADDRESS OF AD SERVER>:49153 ESTABLISHED
TCP <IP ADDRESS OF FUID>:54623 <IP ADDRESS OF AD SERVER>:49153 ESTABLISHED

To check if DC Agent opens new connections, use Windows Services to stop and start DC Agent service.
Based on lab testing DC Agent uses Remote Procedure Call (RPC) to connect to DC server over TCP port
135, and then negotiates dynamic port connection using Endpoint Mapper (EPM).

Additional information about FUID

Here are list of additional sources of FUID knowledge.

Official how to guide for integrating FUID with NGFW:

https://www.websense.com/content/support/library/uid/v11/install/ht_integrate-user-id-service_a_en-us.pdf

FUID 1.1.0 release notes:

https://www.websense.com/content/support/library/uid/v11/relnote/uid_110_rn_a_en-us.pdf

How to configure Forcepoint User ID Service to use a certificate signed by a custom Certificate Authority KB
article:
https://support.forcepoint.com/KBArticle?id=000014489

Integrate Forcepoint User ID Service with Forcepoint NGFW section from SMC 6.4.4 Online Help:
http://help.stonesoft.com/onlinehelp/StoneGate/SMC/6.4.4/GUID-0EF94896-E148-49F2-B598-
2104A49DC27D.html#GUID-B3276772-816D-4E82-880E-C403AF7BFAB8