SIP Troubleshooting Tools

Nick Kwiatkowski – Michigan State University

October 2nd, 2019

Presented by:
Webinar Agenda
• Housekeeping
• Typical SIP Architecture in an Aura environment
• Avaya Aura System Manager (SMGR)
• Avaya Session Manager (ASM)
• Avaya Communication Manager
• Avaya Session Border Controller
• Other ways to troubleshoot
• Q&A

Presented by:
Housekeeping
• Please use the Chat window to ask questions.
• Moderators will be watching the chat, but webinar leader
may elect to push certain questions to the end of the hour.
• Today's webinar will be recorded and available for on-demand
by end of day.
• A short evaluation will pop up as you exit the webinar.
• Please take a minute to provide your feedback about
today's session and what you may like to see going
forward!

Presented by:
About the Instructor
• Manager, Unified Communications Team at
Michigan State University
• Adj. Professor, College of Communication Arts and
Sciences, Michigan State University
• Founding Board Member, Lansing Makers Network
• Apache Software Foundation, Member, Committer
and PMC.
• Avid biker, photographer

Presented by:
Troubleshooting Tools
• This presentation will cover technologies in a typical Avaya
Communication Manager / SIP based setup.
• Will be showing a 8.x system, but everything demonstrated
will be available for 6.1 and beyond. We will mention
major differences when possible.
• Avaya Communication Manager
• Avaya System Manager
• Avaya Session Manager
• Avaya Session Border Controller (for Enterprise)

Presented by:
Troubleshooting
• Remember when
troubleshooting used to be
“easy?”
• One tool was available,
and it gave you a mostly
full view of every
decision that your PBX
was making to route a
call, assign media,
signaling, etc.
• list trace
• There were also MSTs… but
not really useful for
customers..

Presented by:
Troubleshooting
• As the world has moved to SIP, CM is no longer the center of
the Universe…
• CM becomes a feature server,
so it can only see things that
are sent to it. So, in order
to get a more complete
picture, we need to
troubleshoot at the device
that has a view of what we
want to see.

Presented by:
Typical architecture in a SIP environment

Presented by:
Typical architecture in a SIP environment

Presented by:
Typical architecture in a SIP environment

Presented by:
Troubleshooting tools in
Avaya System Manager

Presented by:
SMGR Tools
• Avaya System Manager is the primary database for multiple
servers and services in the environment.
• It does not do any real-time processing. It won’t see any
live traffic, but it does get historical data.
• Since it controls the configs and databases, it can emulate
some decisions that would happen on the system.
• SMGR Can:
• Tell you what devices are registered to your Session
Managers
• Tell you routing decisions that will happen, based on an
input criteria
• Tell you historical data, such as system load of ASMs.

Presented by:
SMGR Tools
• Call Counts
• Session Manager ->
Performance -> Call Counts
• Can tell you the number
INVITE messages sent to
particular SIP entity over a
given time span.
• Useful to gauge trunk
utilization, resources on an
Adjunct. Overloading of
systems or routing loops.

Presented by:
SMGR Tools
• Location Performance
• Session Manager ->
Performance ->
Location Performance
• Tells you the number
of calls terminated in
a location (SIP
location) over a given
period of time.
• Similar info as Call
Counts. Can also
show CACs, if
configured.
Presented by:
SMGR Tools
• SIP Performance
• Session Manager ->
Performance -> SIP
Performance
• Gives you info on a particular
ASM’s view of performance,
either globally, by entity or by
a particular IP.
• Can give you trend data for
traffic or errors to a particular
adjunct.

Presented by:
SMGR Tools
• Tracer Tools
• Session Manager –>
System Tools -> SIP
Tracer Config
• Tracer tools allows
you to automatically
collect SIP traces
that meet a certain
criteria.

Presented by:
SMGR Tools
• Tracer Tools (Continued)
• Tool is greatly improved in SMGR 8.1
• Allows collection of calls based on filters, not just SIP
Entities.
• When a call is triggered against a particular filter, the SIP
trace can be saved (sent to SMGR), or a SNMP trap can be
sent.
• Use cases include sending SNMP traps for 911 calls, toll
fraud, international calls. Can also be used to collect data
for certain scenarios (for example, every time a ‘VIP’ calls,
you can get the trace if something is wrong.

Presented by:
SMGR Tools
• Call Routing Test
• Session Manager –> System Tools -> Call Routing Test

• The call routing test is a tool that allows you to see the
decisions that a Session Manager will make to route a call (an
INVITE statement).
• Tells you final destination based on certain criteria.
• Tells you why it made that decision.
• Includes both routing engines (User and Routed)

Presented by:
SMGR Tools
• Call Routing Test

Presented by:
SMGR Tools

Presented by:
SMGR Tools
• User Registrations
• Session Manager –> System Status -> User Registrations

• User registrations shows the currently registered devices on

any Session Manager managed by that System Manager
• Shows details such as IP addresses, MAC addresses, Models,
event subscriptions, registration history, AST registration, and
which servers a device is registered against.

Presented by:
SMGR Tools

Presented by:
Troubleshooting tools in
Avaya Session Manager

Presented by:
Session Manager Tools
• Avaya Session Manager processes SIP packets from one
endpoint to another. It also handles SIP registration of
endpoints.
• ASM itself has no UI and is administered via SMGR.
• There are command-line (CLI) tools available to see realtime
what is happening on the ASM.
• Remember, ASM can only see data that flows through it….

Presented by:
Session Manager Tools
• traceSM
• The traceSM tool is available via SSH. Requires a customer
or Avaya login into the shell to execute.
• Requires the use of an SSH client. I recommend
SecureCRT or MobaXTerm (Avaya recommends PuTTY)
• To Access, fire up your SSH client and login
• For the base tool, type in traceSM (caps matter).

Presented by:
Session Manager Tools

Presented by:
Session Manager Tools
• When you start the traceSM tool, you will get a few functions
on the bottom of the screen. Important ones to know:
• [F] = Filter. Set up your “display” filters, so you can be
precise in what you are seeing. (Does not work for call
display)
• [Q] = Quit. Eventually you will want to give up.
• [R] = RTP view. Adds a line in the SIP trace that shows
what ASM is guessing the RTP path will be.
• [D] = Call view. Shows all calls in a grid form on ASM
• [W] = Write what is in buffer to disk (think: save-as)
• [S] = Start collection based on current filters.

Presented by:
Session Manager Tools

Presented by:
Session Manager Tools
• Filtering:
• This allows you to filter your current view. Non-
destructive.
• Common filters
• -u (filters on the INVITE/TO/FROM). Any number, user.
• -i (filters on IP address FROM/TO)
• -g (filters on a header packet)
• You can add multiple filters by doing the pipe |. Add
multiple filter types by just adding them in.
• -u 22528|47999 –i 172.24.2.89
• Looks for user 22528 OR 47999 AND the IP address

Presented by:
Session Manager Tools

Presented by:
Session Manager Tools
• When starting a capture, you have a few options.
• SIP Packets – Most likely what you are looking for.
Captures data from the “SM100” module
• PPM Data – Captures PPM, or Avaya addon data. These
are HTTP/HTTPS calls to the ASM server. Button data, etc.
• Call Processing – This shows decision data, similar to
SMGR on how a call is routed. WILL SLOW DOWN YOUR
SYSTEM PROCESSING SIP TRAFFIC!
• TLS Handshake – Shows every TLS handshake and
errored handshakes on the system on every port enabled
for TLS

Presented by:
Session Manager Tools

Presented by:
Session Manager Tools

Presented by:
Session Manager Tools

Presented by:
Session Manager Tools
• RTP Mode will read SIP packets in your current view, and will
“visually” show you where RTP (media) packets are flowing.
They will show within the SIP ladder diagram.
• It reads the SDP (Media Descriptor within SIP) information
that it comes across. Don’t count on it being 100%
accurate.
• It should show the most obvious codec chosen. Codecs
will be negotiated between devices, so another may
actually be used.

Presented by:
Session Manager Tools

Presented by:
Session Manager Tools
• If you find something that you want to review later, or send to
another person (SIP vendor, BP, Avaya, etc), you can save the
current buffer.
• You have the choice to only save data displayed on the
current screen, or ALL the data in the buffer (since the
start of the session or the last 10,000 packets).
• Data is zipped up as a .tgz file, and includes a dump of the
SIP traffic AND a Wireshark compatible file.
• ONLY contains SIP traffic. Remember, media is not seen by
ASM.
• Make sure the name you give is UNIX compatible. No
spaces, slashes, ampersands, etc.

Presented by:
Session Manager Tools

Presented by:
Session Manager Tools
• Call Display View is useful if you are monitoring general traffic
on the switch and want a global view.
• You know a call is coming in, but don’t know the specifics
to filter.
• Things like routing loops will be apparent here.
• The display updates in realtime, showing the progress of calls
on the display. You can press the Enter key on any entry to
see the SIP ladder diagram of the call.

Presented by:
Session Manager Tools

Presented by:
Session Manager Tools
• After you’ve exited the tool by Clearing the buffer, then
pressing Q to quit, you can download or view the saved
sessions.
• Outputs from the [W] command are saved to the user’s
current working directory. Usually /home/cust/.
• If you are using SecureCRT or MobaXTerm, you can
download the file from your current session. Other
terminals you might need to use WinSCP.
• You can open the saved session by appending the name to
the traceSM command.
• traceSM mysavedfile.tgz

Presented by:
 Troubleshooting tools in
Communication Manager

Presented by:
Communication Manager Tools
• Luckily, the tools are the same as you’ve been using for years
in CM… list trace !
• Remember, CM is the application in your network that will be
managing the media codecs in your endpoint connections.
• If you want to see all the SIP traffic CM sees in regard to a
station (or trunk), use the /s parameter. This will show the
regular trace + extra SIP info
• list trace station 22528/s

Presented by:
Communication Manager Tools

Presented by:
Troubleshooting tools in the
Session Border Controller
(for Enterprise)

Presented by:
SBC(e) Tools
• There will be two interfaces you will need to be familiar with
on Avaya’s SBC (for Enterprise).
• The EMS – This is the management tool.
• The SBC Appliance – This is the actual device that
processes SIP traffic.
• Some installs may have these combined onto a single
server.

Presented by:
SBC(e) Tools
• For the EMS, there are a BUNCH of useful troubleshooting
tools at your disposal. In particular, a SIP capture tool.
• EMS -> Device Specific
Settings -> Troubleshooting
Trace
• Set a capture interface, and
up to 10,000 packets.
• Will automatically capture
and send to EMS for you to
download and view in
WireShark.

Presented by:
SBC(e) Tools
• Tip : Finding the Interface:
• EMS -> Device Specific Settings -> Network Management -
> Networks
• Will show all IPs on an interface and the interface name.

Presented by:
SBC(e) Tools
• Other useful tool in the EMS is the list of registered endpoints.
• Status Menu (Top of screen) -> User Registrations
• View remote IP address, see registration is current, see
firmware and device model information.

Presented by:
SBC(e) Tools
• Directly on the SBC device, there is one tool in particular that
is useful -- traceSBC
• traceSBC is exactly like traceSM., with the following
changes:
• It requires root to run.
• It allows you to enter “capture” filters via the command
line
• Capture filters are different than display filters. Non-
captured packets are not available, and they do not
count against the 10,000 packet limit.

Presented by:
SBC(e) Tools
• To do a traceSBC capture:
• Login to the SBC management IP address as ipsec
• su
• traceSBC

Presented by:
SBC(e) Tools
• Capture filters – same as filter dialog box, just on command line.
• traceSBC –u 22528
• The SBC automatically captures traffic for a short amount of time
(busy SBC, this could be hours, could be days)
• SIP logs are stored in /archive/log/tracesbc/tracesbc_sip
• cd /archive/log/tracesbc/tracesbc_sip/
• ls –latr  pick the log of a time you want to view.
• traceSBC /archive/log/tracesbc/tracesbc_sip/date.tgz

Presented by:
SBC(e) Tools
• Once you have a log file open, you can use display filters in
the traceSBC tool like the system was live.

Presented by:
SBC(e) Tools
• The other tool that is useful is tcpdump
• tcpdump will take in all data off an interface and save to a
file, in a format compatible with WireShark.
• If you use tcpdump, make sure to dump to the /archive/
directory.
• tcpdump –i A2 –f /archive/mycapture.pcap
• chown ipcs:ipcs /archive/mycapture.pcap

Presented by:
Other Troubleshooting Tools

Presented by:
Other Troubleshooting Tools
• Network Capture Tools
• Wireshark is the obvious choice.
• Mirror a network port, capture, and view SIP data
• Open a .pcap file from another capture point (like the
SBCE or ASM)
• Network Performance Tools
• IR Prognosis, Oracle Enterprise Operations Monitor, etc.
• Avaya SLAmon Tool (included if you have maintenance)

Presented by:
Other Troubleshooting Tools

Presented by:
Q&A
(Use the Chat Window to ask questions)

Presented by:
Presented by: