FTD Upgrade Troubleshooting

1. Make sure the upgrade package was pushed to FTD:

root@FTD4140:/ngfw/var/sf/updates# ls -al
-rw-rw-r-- 1 root root 942 May 21 13:06 Cisco_FTD_SSP_Upgrade-6.2.0-362.1495371945.info
-rw-rw-r-- 1 root root 246 May 21 13:05 Cisco_FTD_SSP_Upgrade-6.2.0-362.1495371945.lsm
-rw-r--r-- 1 root root 771018529 May 21 13:05 Cisco_FTD_SSP_Upgrade-6.2.0-362.sh

2. Navigate to /ngfw/var/log/sf/upgrade_package directory and check the last lines of

status.log file:
root@FTD4140:/ngfw/var/log/sf/Cisco_FTD_SSP_Upgrade-6.2.0# tail status.log
ui:[23%] Fatal error: Error running script 200_pre/200_enable_maintenance_mode.pl

3. Note the name of the specific script that failed and navigate to the script folder. Check
the contents of the failed script log file:
root@FTD4140:/ngfw/var/log/sf/Cisco_FTD_SSP_Upgrade-6.2.0# cd 200_pre
root@FTD4140:/ngfw/var/log/sf/Cisco_FTD_SSP_Upgrade-6.2.0/200_pre# cat 200_enable_maintenance_mode.pl.log
 FTD Licensing
Case study 1 – Registration failure
• A user tries to register FMC to Cisco Smart Licensing infrastructure, but registration fails

• Recommended Actions – Check DNS and NTP configuration on FMC and verify
connectivity between FMC and Cisco Licensing server (tools.cisco.com) over TCP port
443
• On FMC use ‘pigtail’ and ‘tcpdump’ to verify the background operation
root@FMC:~# tcpdump -i eth0 “host 72.163.4.38 or host 173.37.145.8” –s 1600 –w CAP.cap
FTD CPU usage
FTD provides 2 levels of CPU usage:
• System level
> system support utilization
top - 13:00:15 up 2 days, 3 min, 0 users, load average: 24.29, 24.17, 24.15
Tasks: 719 total, 1 running, 717 sleeping, 0 stopped, 1 zombie
Cpu(s): 29.8%us, 4.9%sy, 0.0%ni, 65.2%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
...
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
13672 root 0 -20 27.2g 2.5g 2.0g S 1794 1.0 51978:46 lina Expected behavior!
13863 root 20 0 5778m 72m 15m S 2 0.0 49:11.73 SFDataCorrelator
> show cpu system
Time CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
19:53:46 all 29.79 0.00 4.92 0.04 0.00 0.01 0.00 0.00 0.00 65.24

• ASA engine
> show level
cpu usage
CPU utilization for 5 seconds = 2%; 1 minute: 1%; 5 minutes: 0%
> show processes cpu-usage sorted non-zero
PC Thread 5Sec 1Min 5Min Process
- - 0.1% 0.1% 0.1% DATAPATH-0-5729
 FTD CPU usage
• To identify the core a Snort Instance runs
> pmtool show affinity
Process CPU Affinity:
...
CPU 2:
e48e2978-3572-11e7-964c-d7923f0c-d01(/ngfw/var/sf/detection_engines/
e48e2978-3572-11e7-
964c-d7923f0c/snort) (2, desired: 2)

• To check the performance statistics for a specific Snort instance

> show perfstats
Available DEs:
1 - Primary Detection Engine (c1400618-3571-11e7-a216-cc613f0c)
0 - Cancel and return to CLI
Select a DE to profile: 1
Available now files:
...
9 - /var/sf/detection_engines/e48e2978-3572-11e7-964c-d7923f0c/instance-1/now
Select a now file: 9 The Snort Instance ID
Thu May 18 00:05:00 2017
Pkts Recv: 1511
Pkts Drop: 0
Mbits/Sec: 1.2
 FTD CPU usage
• To see Min and Max utilization for all Snort instances:
root@FTD:/var/sf/detection_engines/e48e2978-3572-11e7-964c-d7923f0c# for i in `ls ./ | grep "instance-" `; do
perfstats -q < $i/now | egrep "Average|Pkts Recv|Pkts Drop|Mbits/Sec:|Drop Rate|Avg Bytes/Pkt:|instance"; done;

Average Min Max

Pkts Recv : 0 0 0
Pkts Drop : 0 0 0 Snort Instance 1
Mbits/Sec: 0.0 0.0 0.0
Drop Rate: 0% 0% 0%
Avg Bytes/Pkt: 0.0 0.0 0.0
Average Min Max
Pkts Recv : 0 0 0
Pkts Drop : 0 0 0
Mbits/Sec: 0.0 0.0 0.0
Drop Rate: 0% 0% 0%
Avg Bytes/Pkt: 0.0 0.0 0.0
Average Min Max
Pkts Recv : 0 0 0
Pkts Drop : 0 0 0
Mbits/Sec: 0.0 0.0 0.0
Drop Rate: 0% 0% 0%
Avg Bytes/Pkt: 0.0 0.0 0.0
 FTD Memory usage
• Baseline your total memory utilization
• FTD pre-allocates memory for internal usage (~30-40% on ASA appliances, ~20-25% on
Firepower appliances)
• Memory problems can manifest as oom-killer system events (/ngfw/var/log/messages
file):
Aug 18 18:38:27 IDALB-CRPWAN01 kernel: snort invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0
Aug 18 18:38:37 IDALB-CRPWAN01 kernel: nfmex:l3ex invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0

• ASA Engine Access Control List element expansion can affect the memory usage
• Asymmetric traffic can affect system memory usage on NGIPS (Inline-pair) interfaces
FTD Logging – System logs
• Files like /ngfw/var/log/messages contain valuable information about FTD Control and
Data-Plane status
• You can use ‘system support view-files’ command from CLISH to see the contents of
files under /ngfw/var/log directory
> system support view-files
===View Logs===
============================
Directory: /ngfw/var/log
----------sub-dirs----------
packages
removed_packages
...
-----------files------------
2017-04-23 13:09:14.360532 | 3148401 | messages
([b] to go back or [s] to select a file to view, [Ctrl+C] to exit)
Type a sub-dir name to list its contents: s Type ‘s’ to select a file
Type the name of the file to view ([b] to go back, [Ctrl+C] to exit)
> messages Specify the file name
Apr 23 04:02:14 Firepower-module1 CROND[54253]: pam_unix(crond:session): session closed
Snort Restart vs Snort Reload
During a Policy Deployment Snort can either Reload or Restart
Snort No traffic outage during deployment
Reload root@Firepower-module1:/opt/bootcli/cisco/cli/bin# pidof snort
24410 24409 24408 24407 24406 24405 24404 24403
root@Firepower-module1:/opt/bootcli/cisco/cli/bin# tail -f /ngfw/var/log/messages | grep -i 'restart\|reload\|init'
Firepower-module1 SF-IMS[13247]: [13355] ndclientd:ndclientd [INFO] [snort] Received a signal of the start of snort restart
Firepower-module1 SF-IMS[24375]: --== Reloading Snort ==--
Firepower-module1 SF-IMS[24400]: --== Reload Complete ==--
Firepower-module1 SF-IMS[13247]: [13355] ndclientd:ndclientd [INFO] [snort] Received a signal of the end of snort restart
root@Firepower-module1:/opt/bootcli/cisco/cli/bin# pidof snort
24410 24409 24408 24407 24406 24405 24404 24403
Snort Process ID does not change after deployment

root@Firepower-module1:/opt/bootcli/cisco/cli/bin# pidof snort

70243 70242 70241 70240 70239 70238 70237 70236
Snort Traffic outage during deployment
root@Firepower-module1:/opt/bootcli/cisco/cli/bin# tail -f /ngfw/var/log/messages | grep -i 'restart\|reload\|init'
Firepower-module1 SF-IMS[13247]: [13355] ndclientd:ndclientd [INFO] [snort] Received a signal of the start of snort restart
Restart Firepower-module1 SF-IMS[70208]: --== Reloading Snort ==--
Firepower-module1 SF-IMS[70210]: Snort Reload: Changing the so rule memcap configuration requires a restart.
Firepower-module1 snort[72436]: --== Initializing Snort ==--
Firepower-module1 SF-IMS[72464]: --== Initialization Complete ==--
Firepower-module1 SF-IMS[13247]: [13355] ndclientd:ndclientd [INFO] [snort] Received a signal of the end of snort restart
root@Firepower-module1:/opt/bootcli/cisco/cli/bin# pidof snort
32049 32048 32047 32046 32045 32044 32043 32042
Snort Process ID changes after deployment