NSO Troubleshooting

Ramkumar Rajagopalan, ramkraja@cisco.com

NSO Developer days, Stockholm

(c) 2017 Cisco.

NSO Logs and Traces
NSO Logs - ncs.log

• NSO’s own daemon log

• Logs the loading of packages and the starting and
stopping of various components in NSO
• Useful for problems in booting of NSO, reloading
packages or starting the Java-vm etc
• Enabled by default
NSO Logs - developer-log

• Debug log for troubleshooting transactions

• Logs the different stages of transactions, including service
callbacks
• Useful for analysing transaction timings, CDB lock issues
etc.
• developer-log-level => (error | info | trace)
 NSO Logs - developer-log

<DEBUG> 5-Jun-2017::17:58:29.054 RAMKRAJA ncs[16616]: devel-c service_create request for callpoint ‘demo’ path /service:test{1}

CRIT devel-c
ERR Date hostname OS pid devel-cdb Message
WARNING devel-aaa
INFO
devel-snmpa
DEBUG
devel-econfd
…
NSO Logs - north-bound logs

• audit-log, netconf-log, snmp-log, webui-access-log etc.

• Protocol / interface specific logs
• Useful for debugging protocol/interface specific issues
• The audit-log is an audit trail of events in the system
• Most of these are enabled by default
NSO Logs - xpath-trace-log

• Logs all the xpath evaluations done by NSO

• So, when/must expressions in yang models, leafref
checks etc.
• Useful for evaluating transaction performance and yang
validation errors
• Will slow down NSO => disable in production
NSO Logs - error-log

• Contains the internal logs from NSO

• Useful for debugging the NSO daemon itself
• Always good to have this enabled
• Binary file. Can be read by the command
$ ncs --printlog ncs-error.log
NSO Logs - java-vm-log

• Logs from the NSO java-vm and java components in

packages
• Very useful to debug service code
• Not in ncs.conf. Configured under /java-vm in the CDB

admin@ncs% set java-vm java-logging logger com.tailf.maapi

level level-debug
NSO Logs - southbound logs

• Trace between NED and the device

• Configured under /devices/global-settings/trace or
/devices/device/trace in CDB
• Shows the exact commands / RPC’s sent to the device
NSO Logs - CLI NED trace (cisco-ios dry-run)
>> 5-Jun-2017::17:31:39.266 CLI CONNECT to ce0-
127.0.0.1:10022 as admin (Trace=true)
<< 5-Jun-2017::17:31:39.667 CONNECTED 3
>> 5-Jun-2017::17:31:39.669 IS_ALIVE 3
<< 5-Jun-2017::17:31:39.671 IS_ALIVE true
>> 5-Jun-2017::17:31:39.671 PREPARE DRY 3:
(…CLI commands here…)
<< 5-Jun-2017::17:31:39.672 PREPARE DRY
>> 5-Jun-2017::17:32:10.376 CLOSE 3: (Pool: discard)
<< 5-Jun-2017::17:32:10.379 CLOSED
NSO Logs - CLI NED trace (cisco-ios, commit)
>> 5-Jun-2017::17:45:59.507 INITIALIZE 1
<< 5-Jun-2017::17:45:59.616 INITIALIZED (current trans-id)
>> 5-Jun-2017::17:45:59.619 PREPARE 1:
<< 5-Jun-2017::17:46:00.012 PREPARE OK
>> 5-Jun-2017::17:46:00.016 COMMIT 1: (Timeout 30)
<< 5-Jun-2017::17:46:00.025 COMMIT OK
>> 5-Jun-2017::17:46:00.092 PERSIST 1:
<< 5-Jun-2017::17:46:00.093 PERSIST OK
>> 5-Jun-2017::17:46:00.107 GET_TRANS_ID
<< 5-Jun-2017::17:46:00.249 TRANS_ID (new trans-id)
NSO Logs - ncs-collect-tech-report

• Bash script shipped with NSO

• Collect all log files (including, optionally, a CDB backup
and several debug-dumps)
• Only works in a system-install
• Easy way to share logs with the TAC/support teams
NSO Logs - ncs-collect-tech-report
root@demo:/# ncs-collect-tech-report --full --num-debug-dumps 1
Describe the symptoms. In short, what did you do, and what did you see
End by Ctrl-D
some random problem...
Copying log files ...
Taking backup ..
Generating debug dump ...
Show package info ..
tar result ...
Report resides in ncs-tech-report-2017-06-07-20-11.tar.gz
Tools for debugging NSO
ncs --status

Lots of status information about the NSO daemon

• The daemon status
• Loaded yang modules and MIBs
• Loaded packages
• Active user sessions and transactions
• Locks taken on the CDB
• Status of servicepoints, actionpoints, validationpoints
• Clients connected to the CDB (subscribers etc)
NSO debug-dump

• Captures the current internal state of NSO

• Also a binary file
• To get a debug-dump, use
$ ncs --debug-dump dbg.bin
• To read an existing debug-dump, use
$ ncs --show-debug-dump dbg.bin
NSO cdb-debug-dump

• Provides a textual representation of the entire CDB’s

contents
• Does not need a running NSO daemon
• To get a dump of the contents of some set of CDB files:
$ ncs --cdb-debug-dump <path_to_cdb_dir>
NSO utility commands

• ncs_load - Command line utility to load and save NSO

configurations
• ncs_cmd - Command line utility that provides a wrapper to
many common CDB and MAAPI functions
• See Also:
• man ncs_load, man ncs_cmd
NSO CLI Devtools

• CLI-only set of commands to aid during

development/debugging
• Enabled from operational mode, using
admin@ncs> set devtools true
• Used from configure mode
NSO CLI Devtools

• timecmd - Time taken for a CLI command to complete

admin@ncs% timecmd commit dry-run

• Can be used from operational mode as well

admin@ncs> timecmd request devices sync-from
NSO CLI Devtools

xpath - Evaluate xpath expressions from a given context

admin@ncs% xpath ctx devices eval device[port="830"]/name

/ncs:devices/ncs:device[ncs:name='ce0']/ncs:name :: ce0
/ncs:devices/ncs:device[ncs:name='ce2']/ncs:name :: ce2

admin@ncs% xpath ctx devices must device[name="ce0"]/port=830

true
NSO CLI Devtools

‘debug’ commands
• Available as pipe commands to ‘commit’
• debug template - Display the template evaluation
• debug service - Display all FASTMAP operations done by
the service
• debug xpath - Display evaluation of xpath (must/when)
expressions in the service model
NSO CLI Devtools - debug template
admin@ncs(config)# commit dry-run| debug template
Evaluating "$PE" (from file "l3vpn-pe.xml", line 4)
Context node: /vpn/l3vpn[name='volvo']
Result: "pe0”
Operation 'merge' on existing node: /devices/device[name='pe0']
(from file "l3vpn-pe.xml", line 4)
The device /devices/device[name='pe0'] does not support
namespace 'urn:ios' for node "vrf" (from file "l3vpn-pe.xml",
line 6)
Skipping...
NSO CLI Devtools - debug template
admin@ncs(config)# commit dry-run| debug service
Service: /vpn/l3vpn[name='volvo']
shared_create /devices/device[name='ce0']/config/ios:policy-
map[name='volvo'], refcount: 1
shared_create /devices/device[name='ce0']/config/ios:policy-
map[name='volvo']/class[name='class-default'], refcount: 1
shared_set /devices/device[name='ce0']/config/ios:policy-
map[name='volvo']/class[name='class-default']/shape/average/bit-
rate: 12000000, refcount: 1
shared_create /devices/device[name='ce0']/config/ios:class-
map[name='BUSINESS-CRITICAL'], refcount: 1
NSO CLI Devtools - debug template
admin@ncs(config)# commit dry-run| debug xpath
5-Jun-2017::17:35:03.308 Evaluating XPath for:
/vpn/l3vpn[name='volvo']/endpoint[id='branch-office1']/ce-
device:
/ncs:devices/ncs:device/ncs:nameget_elem("/l3vpn:vpn/l3vpn{volvo
}/endpoint{branch-office1}/ce-device") = ce1
exists("/ncs:devices/device{ce1}") = true
get_elem("/l3vpn:vpn/l3vpn{volvo}/endpoint{branch-office1}/ce-
device") = ce1
5-Jun-2017::17:35:03.309 XPath for:
/vpn/l3vpn[name='volvo']/endpoint[id='branch-office1']/ce-device
returns true
Micellaneous
Internal Error

• Unhandled error scenario in NSO

• Error log the only recourse
• At the very least, a better error message might help
• Contact TAC/support with the logs
System issues - Memory

• Hard to debug
• Usual symptoms are random restarts of NSO/Java-vm
• When the logs are not of much use, it’s probably a
memory issue J
• Debug-dumps (or just “top”) taken at regular intervals help
System issues - File descriptors

• Symptoms - Failure to create a new session to a device,

failure to open a log file etc.
• Per process limits - ulimit
• ENOSPC errors while the disk is not full? Check the
inodes => “df -i”
• Linux utilities like lsof, netstat very useful
Thank You