GWC Fault Management - (I) CVM12

Carrier VoIP
Nortel Gateway Controller Fault Management

Release: GWC120 Document Revision: 11.03
www.nortel.com
NN10202-911 .
Carrier VoIP Release: GWC120 Publication: NN10202-911 Document release date: 14 October 2009 Copyright 2009 Nortel Networks. All Rights Reserved. While the information in this document is believed to be accurate and reliable, except as otherwise expressly agreed to in writing NORTEL PROVIDES THIS DOCUMENT "AS IS" WITHOUT WARRANTY OR CONDITION OF ANY KIND, EITHER EXPRESS OR IMPLIED. The information and/or products described in this document are subject to change without notice.
Nortel, Nortel Networks, the Nortel logo, and the Globemark are trademarks of Nortel Networks.
All other trademarks are the property of their respective owners.
Contents
New in this release
Features 5 Other changes 5
Introduction
Navigation 7
7 9
Fault management fundamentals

Navigation 9 Fault management strategy for GWC 9 GWC alarms 10 GWC logs 11 Logs and alarms associated with IPSec 13 Tools and utilities overview 13 CS 2000 managers 13 Integrated Element Management System 14 Fault management in a DQoS network 14 DQoS COPS alarm description 16 Troubleshooting DQoS/COPS connection failures GWC card auto-recovery and boot auditing 17 GWC overload 18 Routine maintenance 20 Data integrity audits 24 Restrictions and limitations 24 CS 2000 data integrity audit 25 Line data integrity audit 25 Trunk data integrity audit 27 V5.2 data integrity audit 27 Viewing GWC service alarm history 30 Viewing GWC service alarms 32 Filtering GWC service alarms 34 Viewing GWC platform hardware alarms 36 Performing GWC hardware diagnostics 38
17
Fault monitoring and troubleshooting procedures
29
Carrier VoIP Nortel Gateway Controller Fault Management NN10202-911 11.03 14 October 2009
Copyright 2009 Nortel Networks. All Rights Reserved.
4 Accessing and printing GWC diagnostic results 40 Viewing and interpreting the operational status of a GWC node 41 Verifying the call processing status of a GWC node 45 Viewing switch-wide GWC status 46 Monitoring operational and activity status of GWCs 53 Viewing GWC PM logs 55 Viewing GWC logs in syslog files 56 Accessing the debug log to view GWC auto-image events 60 Variable definitions 61 Viewing GWC auto-image error logs 62 Interpreting GWC card states 64 Diagnosing problems with a GWC card that cannot be booted 70 Performing a GWC line data integrity audit 73 Performing a CICM line data integrity audit 78 Viewing GWC line data integrity audit reports 81 Performing a GWC trunk data integrity audit 90 Viewing GWC trunk data integrity audit reports 92 Performing a CS 2000 data integrity audit 96 Viewing CS 2000 data integrity audit reports 99 Performing a GWC V5.2 data integrity audit 102 Viewing GWC V5.2 data integrity audit reports 104 Viewing or aborting running data integrity audits 107 Setting up protocol trace environment on the CS 2000 Management Tools server 109 Tracing GWC protocol packets 112 Tracing all packets for a GWC 121 Retrieving protocol trace files 123
Fault clearing procedures

Clearing the GWC318 critical alarm manually 126 Clearing the GWC320 Phase 1 SA failure alarm 128 Clearing the GWC320 Phase 2 SA failure alarm 131 Changing the MAC address of a GWC card 134 Restarting or rebooting a GWC card 140 Restarting GWC card services 142 Replacing and re-provisioning a GWC card 144
125
Alarms and logs reference

GWC service alarms 150 GWC V5.2 audit logs 166 IPSec and IKE security logs 170 Kerberos logs 177 GWC security logs 182
149
New in this release

The following sections detail whats new in Nortel Gateway Controller Fault Management ( (NN10202-911)) for release (I)CVM12:
Features
Features (page 5) Other changes (page 5)
There were no feature changes made to this document.
Other changes
There were no other changes made to this document.
6 New in this release
Introduction
This document includes Gateway Controller fault management concepts, procedures, and reference information.
Navigation
Fault management fundamentals (page 9) Fault monitoring and troubleshooting procedures (page 29) Fault clearing procedures (page 125) Alarms and logs reference (page 149)
8 Introduction
Fault management fundamentals

This section contains information about Gateway Controller (GWC) fault management strategy, alarms, logs, and other fault conditions. It describes tools for troubleshooting faults and alarms and provides fault management information specific to a cable solution that uses dynamic quality of service (DQoS). This section also describes routine maintenance activities that you can perform to prevent faults from occurring, as well as various data integrity audits that you can perform to check for defective data.
Navigation
Fault management strategy for GWC (page 9) Tools and utilities overview (page 13) Fault management in a DQoS network (page 14) GWC card auto-recovery and boot auditing (page 17) GWC overload (page 18) Routine maintenance (page 20) Data integrity audits (page 24)
Fault management strategy for GWC

The GWC uses self-testing, automated diagnostics and reporting systems to support maintenance and manage faults. These systems raise alarms and generate log reports when the following types of hardware or software events occur:
a fault or failure condition correction of a fault or failure condition a threshold is crossed and the GWC is operating at a degraded level or has exceeded a defined operating capacity level a condition occurs that is transient or cannot be repaired
10 Fault management fundamentals
GWC alarms
Alarms provide notification that a system hardware or software-related event has occurred that requires attention. Alarms are generated by the GWC or a related component, such as a gateway, when problems or conditions are detected that can change the performance or operating state of a GWC node and its connections. Administration of the network elements requires monitoring for alarms and checking that functions continue without interruption. The GWC is provisioned with a set of predefined alarms installed. You cannot remove or modify these alarms, although you can disable them. By default, all system alarms are enabled. Alarm management for the GWC is separated into two categories: hardware faults and service and application faults. Hardware fault management activities are carried out using the CS 2000 SAM21 Manager. Service and application fault management activities are carried out using the CS 2000 GWC Manager. Fault clearing depends on the timely resolution of alarms. Alarms provide notification of problems or conditions that can change the performance or working state of the GWC, the CS 2000 or other related network components.
Alarm severity codes

Alarm severity codes indicate the impact of events on the GWC or other network elements. There are four levels of alarm severity:
Critical alarm - This severity level indicates that a service affecting condition has occurred and an immediate corrective action is required. For example, a critical alarm occurs when a managed element is out of service and its capability must be restored. Major alarm - This severity level indicates a service affecting condition that requires an urgent corrective action. For example, a major alarm occurs when there is a severe degradation in the capability of the managed element, such as loss of fault tolerance, and its full capability must be restored. Minor alarm - This severity level indicates a non-service affecting fault condition. Corrective action should be taken in order to prevent a more serious fault that might affect service. A minor alarm occurs when an alarm condition exists that does not degrade the capacity of the managed element. Warning Alarm - This severity level indicates the detection of a potential or impending service affecting fault, before there is any significant effect. Action should be taken to further diagnose and
Fault management strategy for GWC
11
correct the problem to prevent it from becoming a more serious service affecting fault. Based on alarm severity, each alarm has a specific color. Critical and major alarms are red, minor alarms are orange and warnings are yellow. For an example of the alarm severity color codes, see the following figure.
Alarm acknowledgement
It is possible to acknowledge or silence existing GWC service related alarms, although any new alarms cannot be silenced. Use the Integrated Element Management System (IEMS) to perform these functions. For information about alarm acknowledgement, see Nortel IEMS Fault Management ( (NN10334-911)).
GWC logs
A log report is a record of a message that your system or component generates whenever a significant event has occurred on the switch, one of its peripherals or a network element such as the GWC. Log reports include status and activity reports, as well as reports on hardware or software faults, test results, changes in state and other temporary events or conditions likely to affect the performance of the system. A system action or a manual action can generate a log report. When software code traps are generated by faults in the software code running on the GWC, service related PM logs are generated by the GWC to the Core. These logs can be accessed using the logutil application at a maintenance and administration position (MAP) terminal. When fault events occur on the GWC, a simple network management protocol (SNMP) trap is sent to the common SNMP agent that resides on the CS 2000 Management Tools server. The trap is logged using the syslog UNIX logger. The text file output of syslog is saved to a default file location on the CS 2000 Management Tools server.
Alarm information is sent to:
the alarm browser in the CS 2000 GWC Manager the Operations Support System (OSS) interface for presentation to an OSS application (e.g. Micro Muse) the CS 2000 Management Tool server syslog storage for logs.
For syslog storage, the alarm is converted into syslog format before storing. It is possible to disable syslog alarm logging to prevent CS 2000 Management Tools alarms (including the GWC alarms) from being written to the customer log files. You may want to avoid the duplication of these alarms if your system is reporting them using another tool. For information about how to configure alarm logging, see CS 2000 Management Tools sections in Nortel ATM/IP Solution-level Fault Management ( (NN10408-900)). Event log information is sent to:
the alarm browser in the CS 2000 GWC Manager the CS 2000 Management Tool server syslog storage for logs.
GWC log information, included in syslog logs found in the /var/log directory on the CS 2000 Management Tools server, can also be forwarded to the customers OSS interface for analysis. The following items must be in place for the GWC logs to be forwarded to the OSS:
The syslog client and the CS 2000 GWC Manager must reside on the same host (typically the CS 2000 Management Tools server). The Solaris log host on the CS 2000 Management Tools server must be configured to accept remote logs from multiple log sources.
For more information about syslog, and for instructions about syslog forwarding in a network containing the Integrated Element Management System (IEMS), see the CS 2000 Management Tools sections in Nortel ATM/IP Solution-level Fault Management ( (NN10408-900)). For more information about how to access the GWC syslog logs, see procedure Viewing GWC logs in syslog files (page 56). For GWC log report descriptions, see Nortel Carrier Voice over IP Fault Management Logs Reference Volume 2 ( (NN10275-909v2)).
Tools and utilities overview
13
Logs and alarms associated with IPSec

Use the following logs and alarms to monitor and manage faults and other events associated with IPSec:
logs GWC309, GWC320 For more information, see Nortel Carrier Voice over IP Fault Management Logs Reference Volume 2 ( (NN10275-909v2)).
Viewing Kerberos logs Viewing IPSec and IKE security logs alarm SA_PERCENTAGE_USAGE (minor) GWC320 alarms (various specific problems)
For alarms information, see section GWC service alarms (page 150).
Tools and utilities overview

There are three interfaces that you can use to manage faults that occur on the GWC:
Use the maintenance and administration position (MAP) terminal to access the logutil application on the Core to retrieve PM logs. If the fault is related to a service that the GWC performs, such as a trunk or line service, use the CS 2000 GWC Manager to clear the fault. If the fault is related to the hardware state of the GWC card, then use the CS 2000 SAM21 Manager to clear the fault.
For information about how to access the CS 2000 GWC Manager and the CS 2000 SAM21 Manager, see CS 2000 Management Tools sections in Nortel ATM/IP Solution-level Administration and Security ( (NN10402-600)).
CS 2000 managers
The SAM21 Shelf Controllers do not associate Non System Slot (NSS) cards, such as GWCs, as mated pairs and do not monitor application redundancy on GWC cards. For example, a hardware failure resulting in the loss of communication between the managers and a GWC card in the node is handled as follows:
The CS 2000 GWC Manager places the card in an "unknown" state and displays a minor alarm. Any service alarms which were raised by the CS 2000 GWC Manager when the GWC card failed are persisted by the alarm manager, and will continue to be displayed until card service is restored.
The Shelf Controller attempts to recover the card and return it to service.
Although no alarm is raised on the CS 2000 SAM21 Manager, logs are generated indicating that a card has failed.
Integrated Element Management System

You can perform many FCAPS activities using the Integrated Element Management System (IEMS). In addition, you can use IEMS to access the CS 2000 GWC Manager and the CS 2000 SAM21 Manager. To launch the CS 2000 GWC Manager or the CS 2000 SAM21 Manager, see the following procedures in Nortel IEMS Fundamentals ( (NN10329-111)).
"Launching GWC Manager" "Launching SAM21 Manager"
If you wish to acknowledge alarms, see Nortel IEMS Fault Management ( (NN10334-911)).
Use of ping and traceroute

A remote ping and traceroute functionality is provided through the IEMS GUI client. This allows users to launch ping and traceroute operations remotely on the GWC and SPFS platforms. Provision of this facility through IEMS avoids any potential problems caused by allowing non-root users access to these powerful commands. Remote command launch allows users to troubleshoot network connectivity problems from a single location using the same user interface. It initiates an operation on a remote platform or device as if the user had logged on to the device and issued the command directly. Ping and traceroute are accessed from the drop-down menu available when a GWC (or SPFS) managed object is right-clicked in the IEMS GUI. The drop-down menu now includes two new items: Launch Remote Ping and Launch Remote TraceRoute. For more information about how to launch ping and traceroute, see Nortel IEMS Fundamentals ( (NN10329-111)).
Fault management in a DQoS network

In a network using dynamic quality of service (DQoS) implemented for a cable solution, there exist TCP connections between the GWCs and cable modem termination system (CMTS) devices used for authorizing allocation of network resources for each call or connection. A DQoS common open policy service (COPS) connection is a TCP/IP connection used to allow the GWC or policy decision point (PDP) to send call authorizations to
Fault management in a DQoS network
15
the CMTS or PEP. If one of these connections should fail, the gateways associated with the CMTS and controlled by the GWC may still be able to make calls. When a dynamic quality of service (DQoS) connection is down between the CS 2000 and a CMTS, the CS 2000 will allow new calls hosted by that CMTS to proceed without DQoS. The behavior of the multimedia terminal adapter (MTA) and CMTS determines whether new calls are attempted using best-effort service or whether they are torn down:
Some MTA vendors allow calls to proceed as data calls (best-effort) and do not send a data-over-cable service interface specification (DOCSIS) authorization block to the CMTS. In this case, the CMTS cannot recognize the call as a voice call and it proceeds without managed quality of service. Other MTA vendors send the DOCSIS authorization block to the CMTS with no authorization key or gate-id. When this happens, the CMTS decides whether or not to allow calls to proceed.
When the DQoS connection is up and the CS 2000 does not receive a DQoS gate-id from the CMTS, emergency calls (for example, 911 calls, calls originated from ELN line, GETS calls) will not be torn down by the CS 2000. These calls will be allowed to proceed using best-effort QoS (that is, non-DQoS). For normal, non-emergency calls if the CS 2000 does not receive a DQoS gate-id from the CMTS, the CS 2000 will tear down a call if the CMTS reports a GATE-SET-ERR of No Gates Available (1), UnKnown GateId (2), Illegal Session Class (3), Subscriber Limit Exceeded (4), Gate Already Set (5), or Other, unspecified error (127). When the CMTS fails to return a DQoS gate-id and reports a GATE-SET-ERR of Missing Required Object (6), Invalid Object (7), or Illegal DS Field Value (8), the CS 2000 will allow normal, non-emergency calls to proceed using best-effort QoS (that is, non-DQoS). This latter set of errors are considered protocol inter-operation errors and typically would not be observed in the field if the 3rd party CMTS vendor has performed the necessary inter-operation testing with the CS 2000 prior to live deployment. Some CMTS devices are capable of terminating more than the 6400 lines supported on a GWC node. It is therefore important that the customer be alerted to any connection failures between the GWC and CMTS devices. Such connection failures will be reported to the CS 2000 GWC Manager alarm panel.
DQoS COPS alarm description

If a CMTS connection fails on a GWC, a major alarm will be raised using an SNMP trap to the alarm manager. The alarm will automatically be cleared in the same manner when the connection is restored. A DQoS connection alarm will be asserted by the GWC node for each of its connections if:
the connection fails 3 or more times during the 15 second alarm reporting interval the connection fails for more than 5 seconds
A DQoS connection alarm is cleared if: the connection failed less than 3 times during the 15 second alarm reporting interval the connection is up and initialized the connection has been removed by provisioning activity
The alarm text displays "DQoS/COPS connection failure" with specific alarm text "DQoS connection <cmts_name> has failed - attempting recovery." Since the GWC automatically attempts to re-establish any connection, the connection may be recovered before the alarm is actually reported. In this case, the alarm is cleared during the next alarm reporting interval (approximately 15 seconds). If a connection cannot be recovered and the CMTS appears to be functioning normally, call Nortel support to investigate the problem. All DQoS connections are managed in the GWC software to remain up at all times. If a connection fails, the GWC automatically recovers the connection by reconnecting to the CMTS. When a connection fails, the connection is retried almost immediately. If the retry fails, retries continue at a fixed interval until the connection is successfully established or until the provisioning is removed. DQoS connection alarms are reported at least every 15 seconds and at most every 30 seconds after the fault is detected. A connection is considered to be in alarm status if it fails 3 or more times within the 15 second reporting window, or if it is down for more than 5 seconds total during the reporting window. A connection failure that occurs between two reporting windows, such that 2 seconds of outage occur in one window and 3 seconds occur in the next window, is reported in the second window. None of these intervals are customer configurable.
GWC card auto-recovery and boot auditing
17
Troubleshooting DQoS/COPS connection failures

In the event of a COPS connection failure that does not quickly recover, perform the following activities:
Verify that the CMTS specified in the alarm is operational and running a DQoS-capable software version. Look for fault indications on the CMTS that may have led to a failure of the DQoS/COPS server on the CMTS. Verify that the PEP server IP address can be pinged from the GWC IP address. This rules out cable cuts and network problems. Look for alarms and logs generated by the CMTS to the CS 2000 GWC Manager alarm browser or the OSS (if applicable to your solution). Verify that the PEP server IP address configured in the CS 2000 GWC Manager is correct. The PEP IP address is normally the address assigned to the Ethernet interface on the CMTS chassis. Verify that the network is functioning between the GWC that raised the alarm and the CMTS specified in the alarm. This can be done using ping, tracert or similar operating system-level networking tools.
If the problem cannot be resolved, contact your next level of support for assistance.
GWC card auto-recovery and boot auditing

In the event of an application failure on a GWC card, the card goes through an auto-recovery sequence to automatically bring the application back into service. When an application failure occurs, the card is "unlocked-enabled" in the CS 2000 SAM21 Manager, but disabled at the card application level in the CS 2000 GWC Manager. There are two stages of the recovery from an application failure:
1. The Motorola firmware on the GWC card performs a network autoboot

of the card, forcing the card to attempt to boot a software image from the CS 2000 Core Manager or Core and Billing Manager (CBM).
2. If the network autoboot fails, the SAM21 shelf controller performs

a boot of the card in a backup attempt to bring the application into service. This boot audit occurs routinely across the entire shelf. For more information, see Nortel SAM21 Shelf Controller Fault Management ( (NN10089-911)). The following figure shows a sample of the auto-recovery progress text displayed in the "History" window.
During the boot audit, the GWC card transitions from "unlocked-enabled" to "locked-disabled" to "unlocked-disabled" to "unlocked-enabled" in the Card States panel. At the same time, text in the History window of the Card States panel displays an "Auto-recovery in Progress" message followed by the boot recovery sequence messages. The GWC boot audit recovery sequence is also captured in the NSS_boot_audit logs on the shelf controller, for example:
Apr 29 19:36:03: Slot 12 (MCPN750-8): Reset SNMP.1.3.6.1.4.1.562.28.0.1.5.1.2.10 Apr 29 19:36:03: Slot 12 (MCPN750-8): Received MAC address: 08003E2D46D8 Apr 29 19:36:03: Slot 12 (MCPN750-8): Attempting to recover board Apr 29 19:36:04: Slot 12 (MCPN750-8): Rebooting board Apr 29 19:36:19:Slot 12 (MCPN750-8): It took 60s to download boot file. Apr 29 19:36:04: Slot 12 (MCPN750-8): FW_FLASH_VALUE=1 Apr 29 19:40:30: Slot 12 (MCPN750-8): Recovery attempt completed
GWC overload
GWC overload causes the system to generate log PM181. The system continues to output the log every 10 seconds while the GWC unit remains in overload.
GWC overload
19
When a GWC goes into overload, some trunks on that node must be busied to off load traffic. Starting in (I)SN09, the post command at MAPCI TTP level is enhanced to provide the facility for manually busying a specific trunk group on an individual GWC. The post command has a post type I with meaning Post in existing set. When a post command with type I is entered, it posts the trunks in the existing post set (see Figure 1 "Example 1: Post the members of group TRUNK_EXAMPLE on GWC32" (page 19)). If no post set exists when the post command with type I is used, the command returns an error message (see Figure 2 "Example 2: No existing post set for post command with type I" (page 20)). After the I option, the only valid options are G and D. The D option used after the I option supports only the digital equipment GWC and SPM. On the GWC, this feature supports trunk types ISUP, PRI, and PTS. The following figures show examples of the post command using post type I.
Figure 1 Example 1: Post the members of group TRUNK_EXAMPLE on GWC32
Figure 2 Example 2: No existing post set for post command with type I
Routine maintenance
To prevent faults from occurring, perform the following routine maintenance activities at the specified time intervals:
Replace the three air filters from the front of the fan sleds on the SAM21 shelf using the following guidelines:
Replace these air filters once every 10,000 hours (approximately

one year and seven weeks) of service.
When replacing the air filters, replace one air filter at a time and do
not leave a fan uninserted for more than one minute. The Nortel part number of one single air filter is A0828397 (non-RoHS) and N0066842 (RoHS).
Inspect the LEDs on all GWC cards in your system to ensure there are no faults and that all cards appear to be functioning properly. Perform this task once weekly. See the following two figures for details. The following LEDs appear on the front of the GWC cards. All references to the MCPN905 card apply also to the European Union Environment Directives (EUED)-compliant version - CPCI6115.
Routine maintenance
21
SPD/LNK (green/yellow; MCPN905 only) - Ethernet link speed and

status; lights green to show 1000 Mbit link, lights yellow to show 10/100 Mbit link, off if no valid link
ACT (green; MCPN905 only) - Ethernet link activity; lights when

the Ethernet link is active
CPU (green) - CPU activity; lights when the cards processor is

active
BFL (yellow) - Board failure; lights when a system failure occurs

on the card
Hot swap status (blue; in handles of MCPN905 card) - Lights when

it is permissible to remove the card from the shelf In addition to the LEDs, there is also an ABT/RST (abort/reset) button on the front of GWC cards. Press this button briefly (for less than three seconds) to abort the CPUs current process. Press and hold this button (for more than three seconds) to reset the card.
Check and secure the cables and connectors for all GWC cards using the following guidelines:
Check the routing of the cables and how they are secured. Ensure that the data and power cables are routed separately. Inspect the integrity of all cabling to ensure there is no frayed
wiring. Perform these tasks once weekly. The following two figures show the connectors and slots on the front of the GWC cards. The Ethernet port is the cards main network connection. The peripheral component interconnect (PCI) mezzanine card expansion slots are currently not used on a GWC card. Consult your Nortel installation personnel for proper maintenance practices.
22 Fault management fundamentals Figure 3 Details: Motorola N905 NSS board
Routine maintenance
23
Figure 4 Details: Motorola N750 NSS board
Data integrity audits

This section contains conceptual information related to the following data integrity audits:
CS 2000 data integrity audit (page 25) Line data integrity audit (page 25) Trunk data integrity audit (page 27) V5.2 data integrity audit (page 27)
You can perform these audits to check for defective data or if you suspect a provisioning problem. You can also schedule data integrity audits (except CICM line audit). If required, see procedure "Configuring a recurring data integrity audit" in Nortel Gateway Controller Configuration ( (NN10205-511)). Section Performing a CS 2000 data integrity audit (page 96)and Viewing CS 2000 data integrity audit reports (page 99) contains procedures for performing and viewing data integrity audits.
Restrictions and limitations
The audit application keeps no record of problems that caused the alarm. Therefore, if a problem is detected during an audit and it is not corrected, an alarm is generated for the same problem the next time the audit runs. For each audit type, only one audit can be in progress at any given time. An in-progress audit blocks all attempts to run any subsequent audit requests. If you run an on-demand audit, and if that audit is still in progress at the start time of a scheduled audit, the scheduled audit will not occur. This limitation does not apply to a CICM line audit, which can only be run on demand.
If the Management Tools server is configured in a cluster mode (two nodes: one active and one inactive), you can access and view audit reports only on the node that was active during the audit. If a switch of activity (SWACT) occurs after the audit, the new active node does not display any reports. To access audit reports, you need to run the audit again or perform another SWACT.
25
CS 2000 data integrity audit

The CS 2000 data integrity audit compares the GWC Manager database with the CS 2000 Core or the Session Server Manager database, or both and flags any mismatches between the databases. The Core is considered to hold the master database. The compares the following data to highlight inconsistencies:
On the CS 2000 Management Tools server:
bearer network fabric type of each GWC node the network instance of each GWC node
On the Core:
the network instance and fabric type contained in the BEARNETS

table
the bearer network type for each GWC node contained in the
SERVRINV table If the audit detects any inconsistencies in this data, the system gives you the option to attempt to repair them. If your network configuration includes the Session Server for SIP Lines functionality, the audit can also compare the IP-VPN(NAT) network zones configuration data in the GWC Manager and the Session Server Manager databases. This data must be consistent to allow the insertion of Media Proxies for SIP lines. For information about the Session Server Lines virtual gateway, see procedure "Associating a Session Server virtual gateway for SIP Lines" in Nortel Gateway Controller Configuration ( (NN10205-511)). For the Session Server Lines configuration information, see Nortel Session Server Lines Fundamentals ( (NN10437-111)). Unlike the line and trunk audits, the CS 2000 data integrity audit does not provide an option to save the audit report to a file on the local disk.
Line data integrity audit

The line data integrity audit checks the line data stored in the following databases, flags any mismatches, and displays the results of the audit:
CS 2000 GWC Manager database (SESM) Session Server (SS) Manager database (SIP lines) Centrex IP Client Manager database (CICM lines) CS 2000 Core database
For a line audit, the system compares the ENDPOINTENTRY area in the CS 2000 GWC Manager database (SESM) with the following tables in the CS 2000 Core database:
DNINV LGRPINV LNINV HUNTMEM (if hunt groups are provisioned) MDNMEM (if MADN groups are provisioned) LTMAP (if ISDN BRI endpoints are provisioned)
SIP lines audit For SIP lines, the system audits and compares the following data stored in the Session Server (SS) Manager database:
the endpoints - compares with the GWC Manager database group directory number (DN) and member (endpoint/line equipment number [LEN]) information - compares with Core DN to LEN mapping - compares with the Core
CICM lines audit For CICM lines, the system audits and compares data the following stored in the CICM database. The system combines this data with the set of endpoints held by CS 2000 Management Tools server and sends it to CICM to perform the audit. The outcome is returned to CS 2000 Management Tools server and put into the line audit reports.
LCC type and M522 extension modules for working lines (table KSETINV) directory number information against EBS set keys (table KSETLINE) non-directory number feature key data (table KSETFEAT) speed call user feature key data (table SCUFEAT)
The CICM line audit can be run only on demand. The audit supports the following functionality:
Report only mode - audit against CICM but no data correction Correction mode - audit against CICM and refresh CICM data where appropriate the ability to select specific CICM nodes (GW/LGRP granularity) or all nodes
27
Single-direction and double-direction audits In a single-direction audit (SESM to Core), for each line in the SESM table, the system queries the Core tables for corresponding lines. For a single GWC, LGRP, or gateway, only the single-direction line audit is implemented. In a double-direction audit, SESM and Core independently query all the lines in their own tables, then the system compares the SESM and Core lines. For all line audits, the system writes the results into two files: one containing a list of valid data and the other containing a list of problem data. The files are stored on the CS 2000 Management Tools server, on the active unit of the GWC node.
Trunk data integrity audit

The trunk data integrity audit compares the information stored in the CS 2000 GWC Manager and the CS 2000 Core databases, flags any mismatches, and displays the results of the audit. During the audit, the system compares the ENDPOINTENTRY area in the CS 2000 GWC Manager database with the following tables in the CS 2000 Core database:
SERVRINV TRKMEM LTMAP TRKSGRP
The system writes the results of the audit into two files: one containing a list of valid data and the other containing a list of problem data. The files are stored on the CS 2000 Management Tools server.
V5.2 data integrity audit

A V5.2 data integrity audit compares data in the following databases and flags any mismatches:
V5.2 interface data stored in the Network View database V5.2 endpoint data stored in the CS 2000 GWC Manger database V5.2 interface data stored in the table GPPTRNSL in the CS 2000 Core database
Unlike the line and trunk audits, the V5.2 data integrity audit does not provide an option to save the audit report to a file on the local disk.
29

This section contains information and procedures associated with monitoring, analyzing, and troubleshooting GWC fault conditions. Procedures described in this chapter are optional and do not require that you perform them in a specific order.
30 Fault monitoring and troubleshooting procedures
Viewing GWC service alarm history

This procedure allows you to query the GWC service-related alarms that already occurred, and permits alarm display filtering based on GWC unit, alarm severity, alarm category and date/time. Use this procedure as a part of scheduled maintenance and as a primary source of fault diagnostic information for GWC services.
Procedure steps
Step 1 Action From the CS2000 Management Tools window menu, select the Fault and Alarm History.
At the CS 2000 GWC Manager client
Review the alarms displayed. The colors to the left of the alarm display provide a visual indication of alarm severity:

3 4 5
yellow - warning orange - minor red - major and critical
Click Refresh to update the alarm list. Click the Next Page button (if applicable) to view more alarms. Select the Advanced Filters: Enable check box to open the Advanced History Filters dialog box and filter alarms based on selected criteria. If the check box is already selected, click the Settings button to open the Advanced History Filters dialog box.
Procedure steps
31
In the View list, select the GWC units to be excluded (filtered). You can press and hold the <Shift> key to select multiple GWC units. Click the Remove > button to place the selected GWC units in the Exclude (filtered) list. Click the Remove All >> button to place all GWC units in the Exclude (filtered) list. If necessary, select GWC units in the Exclude list. Then, click the < Add button to place the selected GWC units in the View list. Click the << Add All button to place all GWC units in the View list.
Deselect the Severity check boxes to exclude an alarm type. Any alarm severities that remain selected will not be filtered. If required, click the Select All button to select all alarm severity check boxes. De-select the alarm Category check boxes to exclude an alarm type. Any alarm categories that remain selected will not be filtered. If required, click the Select All button to select all alarm category check boxes. If you wish to filter according to a specific range of dates, type the date range in the format, yyyy/mm/dd. If you wish to filter according to a specific time frame, type the time frame in the format, hh:mm. After selecting filter criteria, click the Apply button to display selected alarms and keep the Advanced History Filters dialog box open. Click OK to display selected alarms and close the Advanced History Filters dialog box. To exit the Alarm History, click the File menu at the top of the screen and select Close.
--End--
10 11 12
13
Viewing GWC service alarms

Use this procedure to access service-related alarms that are currently active on the GWC application. The Alarm Manager displays alarms as they occur (in real time). This option also permits alarm display filtering based on GWC unit and alarm category. Use this procedure as a primary source for fault diagnostic information related to GWC services.
Procedure steps
Step 1 Action At the CS 2000 Management Tools window, click the Fault menu and select Alarm Manager to open the Alarm Manager window.
From the Alarm Manager window, review the alarms displayed. The colors to the left of the alarm display provide a visual indication of alarm severity:
yellow - warning orange - minor red - major or critical
See section GWC service alarms (page 150) for details about the alarm types displayed, including appropriate actions to diagnose and resolve the alarm condition. 3 4 5 Click Refresh List to update the alarm list. Click the Details button to review specific details about an alarm. Click the Advanced Filters: Enable check box to open the Advanced Filters dialog box and filter alarms based on selected criteria. If the check box is already selected, click the Settings button to open the Advanced Filters dialog box.
Procedure steps
33
In the view list, select the GWC units to be excluded (filtered). You can press and hold the <Shift> key to select multiple GWC units. Click the Remove > button to place the selected GWC units in the Exclude (filtered) list. Click the Remove All >> button to place all GWC units in the Exclude (filtered) list. If required, select GWC units in the Exclude list. Then, click the < Add button to place the selected GWC units in the View (unfiltered) list. Click the << Add All button to place all GWC units in the View (unfiltered) list.
De-select the Alarm Category check boxes to exclude (filter) an alarm type for the GWC units listed in the Exclude box. Any alarm categories that remain selected will be included (will not be filtered) for the GWC units in the Exclude list.
Click the Apply button to display selected alarms and keep the Advanced Filters dialog box open. Click OK to display selected alarms and close the Advanced Filters dialog box. To exit the Alarm Manager, click the File menu at the top of the screen and select Close.
--End--
10
Filtering GWC service alarms

Use this procedure to filter (exclude) GWC service-related alarms so you are not distracted by alarms that are not relevant to your current fault management activities. Also, use this procedure to filter recurring alarms that you are currently addressing. Use this procedure when implementing your fault management alarm strategy or to focus on specific alarms during alarm clearing or diagnostic activities.
Procedure steps
Step 1 Action From the CS2000 Management Tools window, click the Fault menu and select Alarm Manager. The Alarm Manager window opens.
At the bottom of the Alarm Manager window, click the Advanced Filters: Enable check box to open the Advanced Filters dialog box. If the check box is already selected, click the Settings button to open the Advanced Filters dialog box.
Procedure steps
35
In the View list, select the GWC units to be excluded (filtered). You can press and hold the <Shift> key to select multiple GWC units. Click the Remove > button to place the selected GWC units in the Exclude list or click the Remove All >> button to place all GWC units in the Exclude list. If required, select GWC units in the Exclude list and click the < Add button to place them in the View (unfiltered) list. Click the << Add All button to place all GWC units in the View list.
De-select the Alarm Category check boxes to exclude (filter) an alarm type. Any alarm categories that remain selected will be included (will not be filtered) - for the GWC units in the Exclude list. After selecting your filter criteria, click the Apply button to display selected alarms and keep the Advanced Filters dialog box open. Click OK to display selected alarms and close the Advanced Filters dialog box. To exit the Alarm Manager window, click the File menu in the upper left corner of the screen and select Close.
--End--
Viewing GWC platform hardware alarms

This procedure provides access to platform-related alarms, such as communication over Ethernet, operating system resource availability, and hardware faults. Use this procedure as a part of alarm clearing activity at the CS 2000 SAM21 Manager or as a secondary source of diagnostic information for GWC service-related alarms.
Procedure steps
Step 1 Action Right-click the card in an alarm condition and select Card View from the menu. The Card View window opens.
Click the Alarms tab to display the alarm details.
Procedure steps
37
For information about various alarms generated by the SAM21 platform, see Nortel SAM21 Shelf Controller Fault Management ( (NN10089-911)) . For information about individual alarms related to the NSS cards (including the GWC card) in the SAM21 Shelf, see
CS 2000 Management Tools sections in Nortel ATM/IP Solution-level Fault Management ( (NN10408-900)) in wireless markets - Packet MSC Fault Management (NN-20000-212).
--End--
Performing GWC hardware diagnostics

This procedure describes how to perform hardware diagnostics on the GWC card and save the diagnostic results to an ASCII text file for later analysis. Use this procedure as a secondary source of diagnostic information or when a hardware fault persists.
Prerequisites
Before starting this procedure, lock the GWC card on which you wish to perform diagnostics. Follow procedure "Locking a GWC card" in Nortel Gateway Controller Administration and Security ( (NN10213-611)).
Procedure steps
Step 1 Action In the Card View window, click the Diags tab.
At the CS 2000 SAM21 Manager client
2 3
Select Brief or Full diagnostics using the drop-down menu. Click the Start button. If necessary, you can stop a diagnostics test in progress by clicking the Stop button. A diagnostics icon appears on the GWC card in the Shelf View.
Procedure steps
39
Diagnostic messages appear in the Status area of the Card View. 4 If the test passes, click the Save button at the bottom of the Card View window to save the results on the CS 2000 SAM21 Manager client. Choose a name and location for the diagnostics file. Append the file with a ".txt" extension for easy identification. To retrieve and print the results, follow procedure Accessing and printing GWC diagnostic results (page 40). The procedure is complete. If the hardware diagnostic test fails for any reason, go to the next step. 5 Rerun the diagnostics using the Brief test option.
If the Brief test passes continue with the next step. If the Brief test fails go to Step 7.
Rerun the diagnostics using the Full option.
If the Full test passes unlock the card and return it to service.
ATTENTION
If you attempt to unlock a card that failed a diagnostic test, the system displays a message warning you that the card may not be fully functional. If this occurs, rerun the diagnostic test. If the card fails a second time, replace the card and contact your next level of support.
If the Full test fails go to the next step.
Replace the card. Follow procedure Replacing and re-provisioning a GWC card (page 144). Return the defective card to Nortel according to the procedures of your service contract.
--End--
Accessing and printing GWC diagnostic results

Use this procedure to retrieve and print diagnostic results from a saved ASCII text file stored on the SAM21 client workstation.
Prerequisites
Perform a diagnostic test using procedure Performing GWC hardware diagnostics (page 38).
Procedure steps
Step 1 2 3 Action Open a terminal session on the client workstation. Enter: cat </path/to/file/filename> If a printer is available on the network, print a copy of the diagnostic results. Enter: lp -c </path/to/log_file/filename> <printername>
--End--
Variable denitions
Variable </path/to/file/filename> </path/to/log_file/filename> <printername> Value The directory location of the diagnostic file The directory location of the log file The system name of the printer connected or mounted to the CS 2000 SAM21 Manager (if available)
Procedure steps
41
Viewing and interpreting the operational status of a GWC node

Use this procedure to determine the operational status of a selected Gateway Controller (GWC) node using the CS 2000 GWC Manager. Use this procedure as a primary source of information about the operational status of a GWC card or GWC node.
Procedure steps
Step 1 2 3 Action At the CS 2000 Management Tools window, click the Gateway Controller folder from the Device Types menu. From the Contents of: Gateway Controller frame, select the GWC node that you wish to view. Click the Maintenance tab. The GUI displays the Maintenance panel with two independent status views, one for each of the GWC cards in the node.
See Table 1 "CS 2000 GWC Manager status fields" (page 42) following this procedure to interpret the GWC card (unit) status fields.
If the selected GWC loses communication with the GWC Manager, the client does not provide an accurate status of the GWC node. You can verify the call processing status of a GWC node using the MAPCI interface. If required, follow procedure Verifying the call processing status of a GWC node (page 45).
--End--
Job aid
The following table describes the GWC card (unit) status fields.
Table 1 CS 2000 GWC Manager status fields Status field Administrative state: Possible values locked Meaning The unit is prohibited, administratively, from providing service to users.
A status of "locked" on the CS 2000 GWC Manager indicates that the software application on the card is no longer performing its primary call processing function, but the card is still running. (The call processing function has been "busied", but underlying maintenance and communications activities are still functioning.) A status of "locked" on the CS 2000 SAM21 Manager indicates that the hardware is locked to ROM level, and the software application is no longer running. unlocked Operational state: enabled disabled The unit is permitted, administratively, to provide service to users. The unit is partially or fully providing service to users. The unit is not operating or providing service to users. If the Administrative state for this unit is "locked", then the unit has been manually busied. If the Administrative state for this unit is "unlocked", then the unit has been busied by the system. The unit is currently providing end user services. This is the state of the node as seen by other network elements. The unit is not providing end user services but can be switched to Active at any time if the active (mate) unit fails. The unit is not communicating with the Core.
Activity state:
active
standby
Isolation state:
isolated
Job aid
43
Table 1 CS 2000 GWC Manager status fields (contd.) Status field Possible values not isolated Available state: offLine(3) Meaning The unit is communicating with the Core. The unit has not received its configuration data from the CS 2000 GWC Manager. The unit cannot provide service until it is booted and receives configuration data. The unit does not have heartbeat communication with its mate and it is operating without fault-tolerant redundancy. The unit has both: offline and degraded conditions. The unit does not have either of the preceding conditions. This is the name of the load file that the unit currently boots from. The file is located on the CS 2000 Core Manager or Core and Billing Manager (CBM) disk drive. The GWC maintenance system is not currently working on a request, such as a Return to Service (RTS). The unit is available for maintenance requests. Maintenance is in progress on this unit and no further requests are accepted. The unit is the active unit and is providing service. The unit is the standby unit - ready to provide service. The unit is synchronizing with the active unit (not providing redundancy). After completion of synchronization, the status changes to hotStandby when the Operational state is enabled. This field indicates the last switch of activity for the unit. manualSwActWarm Last switch of activity was due to a manual warm SwAct. Requested by a user, a warm SwAct causes no service interruption to stable calls, but calls in the setup processes can be lost.
degraded(6)
offLine(3), degraded(6) 00 00 00 00 Loadname: <string_of_alphanumeric_ characters>
Usage state:
idle
busy Stand by state: providingService hotStandby coldStandby
Swact state:
Table 1 CS 2000 GWC Manager status fields (contd.) Status field Possible values manualSwActCold Meaning Last switch of activity was due to a manual cold SwAct. Requested by a user, a cold SwAct temporarily takes both units out of service and takes down all calls. Last switch of activity was due to a system warm SwAct. These SwActs are automatically performed by the device in response to faults or failures. Established calls are preserved. Calls in setup are lost. Last switch of activity was due to a system cold SwAct. These SwActs are automatically performed by the device in response to faults or failures. All calls are lost. No switch of activity has occurred. This field indicates the severity of the currently raised alarms. 00 00 00 00 critical(1) major(2) minor(3) alarmOutstanding(4) There are no alarms raised on the GWC card unit. If present, indicates that one or more critical alarms have been raised. If present, indicates that one or more major alarms have been raised. If present, indicates that one or more minor alarms have been raised. If present, indicates that at least one or a combination of different alarms has been raised. This field is not used.
autonomousSwActWarm
autonomousSwActCold
noSwAct Alarm state:
Fault state:
none(0)
Variable definitions
45
Verifying the call processing status of a GWC node

If a GWC loses communication with the GWC Manager, the client does not provide an accurate status of the GWC node. In this scenario, use this procedure to verify the call processing status of a GWC node - using the MAPCI interface.
Procedure steps
Step 1 2 Action Access the peripheral module maintenance level. Enter:
MAPCI;MTC;PM
Post the desired GWC node in the control position. Enter: post GWC <node_number> The system displays both GWC units and their current states.
Verify the current state of the selected GWC node. The GWC node can be in one of the following states:
InSv (in service) If one or both units are in an InSv state, the GWC is capable of performing call processing. SysB (system busy) or ManB (manual busy) If both units are in SysB or ManB state, the GWC is not capable of performing call processing.
--End--
Variable denitions
Variable <node_number> Value The node number of the GWC that you want to post.
Viewing switch-wide GWC status

Use this procedure to query switch-wide Gateway Controller (GWC) status information. You can view the status of a specified GWC or all GWCs in your network. If required, you can also store the information in a file on a specified device. Use this procedure when needed, for example, when responding to a service degradation or an outage. Commands used in this procedure are not case-sensitive.
Procedure steps
Step 1 Action Access the GWC status level. Enter:
GWCSTAT
At the CI interface
System response:
GWCSTAT:
This step is optional.
If you wish to query all subcommands available from the GWCSTAT level, enter:
help gwcstat
If you wish to query the usage of a specific subcommand, enter:

help <subcommand>
Use the appropriate subcommand to display the required information associated with a specific GWC or with all GWCs in the network. For each subcommand,
[ ] (square brackets) represent optional parameters; the default output is console, but you can direct the output to a specified file on a specified device, using the optional "file" parameter.
For detailed description of each command, see Table 2 "GWCSTAT commands" (page 50). Enter one of the following commands:
config <ALL or gwc#> [file <device_name> <file_name>]
OR
status <ALL or gwc#> [file <device_name> <file_name>]
OR
Procedure steps alarms <ALL or gwc#> [file <device_name> <file_name>]
47
OR
gwlist <ALL or gwc#> [file <device_name> <file_name>]
OR
all <ALL or gwc#> [file <device_name> <file_name>]
OR
gwctrkquery gwc <ALL or gwc#> [file <device_name> <file_name>]
OR
gwctrkquery clli <clli_name> <member_id_start> [<member_id_end>] [file <device_name> <file_name>]
Example commands and system responses

> config 1
Example system response:
> status all
> gwlist all
> alarms 0
> all all
This command displays the combined output of the following four commands: config, status, gwlist, and alarms.
> gwctrkquery gwc 1
> gwctrkquery clli 0 1
49
If you wish to return to the CI level, enter

QUIT
--End--
Variable denitions
Variable <ALL or gwc#> Value ALL use this parameter if you wish to display status information for all GWCs in the network. OR A number between 0 and 255 indicates which specific GWC you wish to query. For example, for GWC-8, enter 8. Optional parameter. If you wish to direct the output to a file on a specified device, enter file, followed by a device name and a file name. Common language location identifier (CLLI) code of the trunk group that you wish to query. The trunk member numbers that together define the range of trunks within a trunk group that you wish to query. The member_id_start is a mandatory parameter, but member_id_end is optional. Examples If you enter 0 for <member_id_start> and 10 for <member_id_end>, the system displays trunks whose member numbers are between 0 and 10. If you enter the same number for both parameters (for example, 5 5), the system displays the trunk with member number 5). If you enter only the <member_id_start> number, the system displays trunks whose member numbers are equal or greater than that specified number. For example, if you specify 5, the system displays trunks whose member number is equal or greater than 5. <subcommand> One of the following subcommands available from the GWCSTAT level: config, status, gwlist, alarms, all, or gwctrkquery.
[file <device_name> <file_name>] <clli_name> <member_id_start> [member_id_end>]
Job aid
Table 2 GWCSTAT commands Command Description
For all commands listed in this table, if the queried GWC does not respond, the system displays the Not Responding message next to the affected GWC. config Use this command to view the following configuration information for a specified GWC or for all GWCs in the network: element manager (EM) IP address
IP addresses for CORE. GWC number node number active IP address profile name
Active IP, Inactive IP, Unit0 IP and Unit1 IP are sequential, increase by 1, so Inactive IP, Unit0 IP and Unit1 IP are not listed in the output. You can derive them from the Active IP. status Use this command to view the general status information of a specified GWC or all GWCs in the network. This command displays the GWC load name and the following states: AVAILST (available)
FAULT ADMIN (administrative) USAGE OPER (operational) STANDBY ACT (activity) SWACTST (Swact) ISOLST (isolation)
You can also view this information using the GWC Manager. For more information about these states and their possible values, see Viewing and interpreting the operational status of a GWC node.
Job aid
51
Table 2 GWCSTAT commands (contd.) Command gwlist Description Use this command to display the following information about all gateways associated with a specified GWC or with all GWCs in the network: gateway name
gateway IP address gateway type (for example, LARGE_GW) gateway state (DISABLED or ENABLED) protocol name and version (for example, MEGACO 1.0) protocol port number
ATTENTION
For wireline solutions, the following queries are not supported:
gateways associated with a small line GWCs System displays the following message: Small line not supported GWCs with associated RMGC or small line gateways System displays the following message: Gateway type not supported
alarms all gwctrkquery gwc
Use this command to list the number of critical, major, minor, and warning alarms for a specified GWC or all GWCs in the network. Use this command to display the combined output of the following four commands: config, status, gwlist, and alarms. Use this command to display the information about trunks associated with a specified GWC or all GWCs in the network. This command lists the following information: GWC number

gwctrkquery clli
external terminal number (EXTTERMNO) trunk group identifier (CLLI) trunk member number (MEMBERNO) state of the trunk subgroup type (SGRPTYPE)
Use this command to display GWC trunks information for a specified trunk group. This command lists the following information: trunk group clli and trunk member number (Trunk:)
subgroup type (Sgrp Type:) state of the trunk GWC number external terminal number (ExtTermNo:)
Table 2 GWCSTAT commands (contd.) Command Description
gateway name (GW name:) gateway IP address (GW IP:) endpoint Core status (SSC_State:)
Because there can be thousands of endpoints associated to a GWC, the system displays only ten members at a time, then prompts to you continue.
Procedure steps
53
Monitoring operational and activity status of GWCs

Use this procedure to access the GWCALL command under MAPCI;MTC;PM level, which allows you to continuously monitor operational and activity status of all Gateway Controllers (GWC) in the network. The system updates the output whenever the status of any GWC changes.
Procedure steps
Step 1 2 Action Access the GWC directory: Enter:
MAPCI;MTC;PM;POST GWC
At the CI interface
Display the operational and activity status of all GWCs. Enter:

GWCALL
The system displays all GWCs sorted by their number, each row represents a unit of GWCs. The system updates the output whenever the status of any GWC changes. For the description of the symbols used in the display, see Table 3 "GWCALL status symbols" (page 54). Example system response:
If you wish to return to the CI level, enter

QUIT ALL
--End--
Job aid
Table 3 GWCALL status symbols Symbol . (dot) * (star) - (dash) M m S s O o Meaning The GWC unit is active and its operational state is InService. The GWC unit is inactive and its operational status is InService. The GWC unit is unequipped. The GWC unit is active and its operational status is ManB (manually busy). The GWC unit is inactive and its operational status is ManB. The GWC unit is active and its operational status is SysB (system busy). The GWC unit is inactive and its operational status is SysB. The GWC unit is active and its operational status is Offl (off line). The GWC unit is inactive and its operational status is Offl
Procedure steps
55
Viewing GWC PM logs

This procedure describes how to access service-related peripheral module (PM) logs generated by the GWC and forwarded to the Core. Use this procedure as a part of scheduled maintenance and as a secondary source of diagnostic information.
Procedure steps
Step 1 2 Action Enter: logutil To retrieve the latest PM log, enter: open pm
For more information about commands available in logutil, enter: print logutildir For specific information about the logs, and any actions required, see the PM log descriptions in Nortel Carrier Voice over IP Fault Management Logs Reference Volume 2 ( (NN10275-909v2)) .
--End--
Viewing GWC logs in syslog les

This procedure describes how to access GWC logs stored in the syslog files on the CS 2000 Management Tools server by logging into the CS 2000 Management Tools server. You can also access syslog files using the Integrated Element Management System (IEMS) GUI. For more information about viewing the audit and the security logs, see Nortel IEMS Administration and Security (NN10336-611). This procedure also describes how to search for specific entries in the syslog files. Use this procedure as a part of scheduled maintenance and as a secondary source of diagnostic information.
Prerequisites
You need the root user ID and password to log into the CS 2000 Management Tools server.
Procedure steps
Step 1 Action At your workstation, establish a login session to the CS 2000 Management Tools server using one of the following methods:
telnet (unsecure) For instructions, go to step 2. ssh (secure)
ATTENTION
Use this option only if your workstation platform supports ssh. Otherwise, your attempt to log in fails.
For instructions, go to step 7. 2 3 4 5 Enter: telnet <server> When prompted, enter you user ID and password. The system prompt changes to a dollar sign ($). Change to the root user. Enter: su When prompted, enter the root password.
Procedure steps
57
6 7 8 9
Go to step 9 to continue the procedure. Enter: ssh -1 root <server> When prompted, enter the root password. The system prompt changes to a dollar sign ($). At the CS 2000 Management Tools client, access the directory level where the syslog files reside. Enter:
cd /var/log
10
List the directory content. Enter:

ls
The system displays a list of different log files, such as, customerlog, securitylog, and so on. These files are appended with numbers, for example "customerlog.0". The files with the lower numbers are the newer files. For a list of GWC syslog logs, see Job aid (page 58). 11 If the file that you want to view is zipped (has an extension .gz), unzip it using the following command. Otherwise, go to the next step. To unzip the file, enter: gunzip <log_filename.gz> Example
$ gunzip securitylog.1.gz
The file changes to a readable file securitylog.1. 12 Enter this command to display the entire content of a log file: cat <log_filename> |more Example $ cat customerlog.0 |more Press the space bar to scroll through the file if it is larger than the screen can display. OR Enter this command to search and display specific content of a log file: cat <log_filename> |grep <search_string> Examples
cat customerlog.0 |grep GWC309 cat securitylog.1 |grep KRB_LOG
13
If you want to print the file, contact your site system administrator for assistance with using UNIX print commands and with locating a printer connected to your network.
--End--
Variable denitions
Variable <log_filename> <log_filename.gz> <search_string> Value The name of the log file that you wan to display. The name of the log file that you want to unzip. The text that you want to search for. For example, KRB (to search for logs associated with the Kerberos application) The hostname or IP address of the server.
<server>
Job aid
The following tables describe the syslog logs in the /var/log directory that contain entries relevant to the GWC.
Table 4 Syslog logs containing GWC entries Log type Audit log Description Records the actions taken by users on the system, including some of the parameters they used. Records all the user logins and logouts from any source using ftp, proftp, telnet, sftp, scp, or ssh. The successful logins and logouts are logged to auditlog (/var/log/auditlog). Customer log Records all alarms the system has received. customerlog customerlog.0 customerlog.1 Examples of log file names auditlog auditlog.0 auditlog.1
Job aid
59
Table 4 Syslog logs containing GWC entries (contd.) Log type Debug log Description Records debug information for CS 2000 Management Tools network components to help detect an underlying problem. Contains a record of all the SNMP traps received by the system. Examples of log file names debuglog debuglog.0 ptmlog ptmlog.1 ptmlog.2 Security log Records failed actions taken by users on the system. Securitylog file also includes fault-related logs for Kerberos and IKE/IPSec security. Records all the user logins and logouts from any source using ftp, proftp, telnet, sftp, scp, or ssh. The login failures are logged to securitylog (/var/log/securitylog). securitylog securitylog.0
PTM log
Use the following table to interpret the syslog application logs on the redirecting media gateway controller (RMGC).
Table 5 GWC syslog application logs Application log description RMGC: Successful Count: x Failed Count: y Cause or condition The RMGC application produces a syslog performance report once an hour. (See the debug log in /var/log.) It contains the counts of the number of RSIPs processed successfully (x) and the number failed (y). The counts are cumulative, so that to calculate the number of successful/failed RSIPs, it is not necessary to parse each and every log but just to subtract the counts from the previous log to derive the counts between the current log and the previous. Action No action required
Accessing the debug log to view GWC auto-image events

This procedure describes how to display the content of the CS 2000 GWC Manager debug log and search for auto-image events in the log. The debug log resides on the CS 2000 Management Tools server. Use this procedure if you are troubleshooting a problem with auto-imaging and you want to search for a specific auto-image entry in the debug log.
Prerequisites
You must have a user account on the CS 2000 Management Tools server to access the error log located in the following directory: /opt/nortel/NTsesm/admin/logs.
Procedure steps
Step 1 2 3 Action Log onto the server as the root user. Change to the /opt/nortel/NTsesm/admin/logs directory. Enter:
cd /opt/nortel/NTsesm/admin/logs
At the CS 2000 Management Tools client
Open the debug log. Enter: less <debug_log_filename> Example

less ptmdebuglog1.mi2
Example Response
03.01.28 13:30:28.697 VRB (ubsnmp) [PE-8] UBSnmpSimpleT rap Notifying 1 listeners 03.01.28 13:30:28.697 VRB (MI2Server) [PE-8] TrapLogger::trapNotification#queue: 1 03.01.28 13:30:28.699 VRB (EM) [Thread-78] GWCUtils: Attempting to convert gwcid
Search for auto-image events while in the debug log. Enter one of the following commands: /auto /Auto
61
/AUTO The search string you enter is case sensitive. Each search yields different results. The text that you have searched is highlighted in the display. Example
/Auto
Example response
03.01.29 10:31:34.235 VRB (gwcem@1) [RequestProcessor[1]] SesmSecureProxy::invoke isAutoImagingEnabled 03.01.29 10:31:34.236 VRB (gwcem@1) [RequestProcessor[1]] AuthorizingHandler.authorize: isAutoImagingEnabled () 03.01.29 10:31:34.258 NOA (gwcem@1) [RequestProcessor[1]] AUDIT: isAutoImagingEnabled ()
5 6 7
Type n to repeat the previous search in the downwards direction. Type N to repeat the previous search in the reverse direction. Repeat step 4 to search for other entries of the same text in the debug log, or for different text. Type q to exit.
--End--
Variable denitions
Variable <debug_log_filename> Value The name of the debug log file. This file name can be configured. The default file name is ptmdebuglog<n>.mi2, where <n> is a number that increments as the log increases in size.
Viewing GWC auto-image error logs

This procedure describes how to display the contents of the auto-image error log. This log provides a summary of any errors that prevent auto-imaging from taking place. The error log resides on the CS 2000 Management Tools server. The log recycles after recording 2000 lines of text. Use this procedure when you want to determine if auto-imaging is working properly.
Prerequisites
You must have a user account on the CS 2000 Management Tools server to access the error log located on the CS 2000 Management Tools server in the following directory: /opt/nortel/NTsesm/admin/logs.
Procedure steps
Step 1 2 3 Action Log onto the server as the root user. Change to the /opt/nortel/NTsesm/admin/logs directory. Enter:
cd /opt/nortel/NTsesm/admin/logs
At the CS 2000 Management Tools client
Display the content of the auto-image error log. Enter:

more AutoImagingErrorLog
Example response
Auto Imaging Executed Tue Jan 28 02:00:01 EST 2003 An error occurred while auto imaging: Auto imaging has not been enabled. Auto Imaging Executed Wed Jan 29 02:00:01 EST 2003 An error occurred while auto imaging: Auto imaging has not been enabled.
See section Job aid (page 63) for some examples of auto-image logs and suggested actions.
--End--
Job aid
63
Job aid
The following table provides examples of auto-image error logs, information on how to interpret them, and suggested actions. Auto-image logs are recorded in the format: Auto Imaging Executed <day><month><dd><hh:mm:ss><yyyy> "log description"
where day/month/dd/hh:mm:ss/yyyy is the date and time stamp for the log "log description" is a description of the conditions or reasons for generating the log.
Table 6 Examples of auto-image error logs Auto-image log description An error occurred while auto imaging: Auto imaging has not been enabled. Cause or condition An image of the software loads on your GWC devices was not taken automatically at the scheduled time because auto-imaging was not enabled. Action Enable auto-imaging using the CS 2000 GWC Manager if you want to automatically save an up-to-date image of GWC software loads once daily on the CS 2000 Core Manager or Core and Billing Manager (CBM). See procedure "Enabling or disabling GWC software auto-imaging" in Nortel Gateway Controller Configuration ( (NN10205-511)). An error occurred while trying to connect to the GWC EM. The CS 2000 Management Tools server is down. Restart the CS 2000 Management Tools server.
Interpreting GWC card states

This procedure describes how to determine and interpret the GWC card state using the CS 2000 SAM21 Manager. Use this procedure when you are encountering an unknown card state.
Procedure steps
Step 1 Action Review the following figure and determine the card icons that apply.
These states also apply to Shelf Controllers. 2 To view the card state tab, right-click the card icon and select Card View . The Card View window opens. 3 Select the States tab.
Job aid
65
Use Table 7 "GWC card states and possible actions" (page 65) to determine your next action, based on the cards state. A status of "locked" on the CS 2000 SAM21 Manager indicates that the hardware is locked to ROM level, and the software application is no longer running. A status of "locked" on the CS 2000 GWC Manager indicates that the software application on the card is still running, but it is no longer performing its primary call processing function.
--End--
Job aid
This section contains job aids related to this procedure.
GWC cards states

Table 7 GWC card states and possible actions State locked-disabled-offduty Possible action Wait for the firmware flash to complete. Verify that the card changes to the locked-disabled-none state. If the GWC card transitions to locked-disabled-d egraded, follow the suggestions for that state. unlocked-disabled-offduty For Call Agent cards, this state also represents the restart and reload of the call processing application during a routine exercise test (RExTst). When the Shelf Controller performs its boot audit, any GWC card that is not running or booting is set to this state until the Shelf Controller recovers the card.
Table 7 GWC card states and possible actions (contd.) State locked-disabled-offline (new card) Possible action Right-click the card icon and select Assign Service from the card context menu. Select the correct service from the Assign Service window. If the question mark icon does not disappear, open the Card View and view the States tab. If the history text area indicates that service assignment failed because the service type is incompatible with the hardware, either replace the card with the correct hardware type, or unassign service from the shelf view and then assign the correct service type. locked-disabled-offline (reinsertion) Wait for Shelf Controller to recognize the card and reinstate the provisioning information. The question mark icon disappears and the card transitions to a new state. Refer to the suggestions for the new state. If the question mark icon does not disappear, open the Card View window and view the States tab. If the history text area indicates that service assignment failed because the service type is incompatible with the hardware, either replace the card with the correct hardware type, or unassign service from the shelf view and then reassign with the correct service type. If the history text area indicates that the service assignment failed because the IP address is already reserved by another unit, contact network engineering to determine if another unit is misconfigured, or if this unit should be reconfigured.
Job aid
67
Table 7 GWC card states and possible actions (contd.) State locked-disabled-none or locked-disabled-degr aded Possible action Unlock the card by right-clicking on the card icon and select Unlock from the card context menu. Rerun diagnostics if the CS 2000 SAM21 Manager client generates a "Degraded state Unlock confirmation window". If diagnostics fail a second time, replace the card and contact Nortel support personnel. The active Shelf Controller generates two critical alarms when the inactive Shelf Controller is locked. A locked-disabled- degraded state for non system slot (NSS) cards is also alarmed. locked-disabled-failed This card is inaccessible. Verify the following items:
Shelf Controllers are in service If the Shelf Controllers are in service, replace the card. If the replacement card does not enter unlocked-enabled-none, contact Nortel support personnel.
locked-disabled-in test
Wait for diagnostics to complete. Verify that the card changes to the locked-disabled-none state. Optionally monitor diagnostics progress from the Card View window.
unlocked-enabled-degraded
This card failed one or more diagnostics and was Unlocked. See Unlocking a card that has failed a diagnostic test (page 68) at the end of this procedure. This card may not be providing service or may be unreliable. Lock and run diagnostics on this card. If the card fails diagnostics, replace this card and contact Nortel support personnel.
Table 7 GWC card states and possible actions (contd.) State locked-disabled-none and alarmed Possible action This card has taken more than three minutes to complete a lock or unlock request. The alarm clears when the card completes the request or is removed from the shelf.
locked-disabled-failed (no application)
The Shelf Controller detects a card in the slot, but cannot determine the MAC address for the card. Reinsert the card.
Unlocking a card that has failed a diagnostic test

The following message appears on the CS 2000 SAM21 Manager if you attempt to unlock a card that has failed a diagnostic test. If this occurs, rerun the diagnostic test. If the card fails a second time, replace the card and contact Nortel support personnel.
Information card icon

An additional shelf view card icon indicates that the CS 2000 SAM21 Manager client cannot display all the card icons. Click this information icon to view the card state information in a balloon. This icon normally indicates that the card type is not supported for the current release of the CS 2000 SAM21 Manager software.
Job aid
69
Line integrity data audit is design intent. During the audit, avoid doing any maintenance actions, such as GWC status query or GWC swact, and any configuration actions, such as add gwc and delete gwc, because sometimes the bearer network cannot be retrieved due to response timeout from ddms.
Diagnosing problems with a GWC card that cannot be booted

Use this procedure to diagnose problems with a GWC card that is installed in the SAM21 shelf but does not appear on the CS 2000 SAM21 Manager shelf view and it cannot be booted.
Prerequisites
You must have root user access to the CS 2000 Core Manager or Core and Billing Manager (CBM) console.
Procedure steps
Step 1 2 Action Login to the CS 2000 Core Manager or CBM as the root user. Start the CS 2000 Core Manager or CBM maintenance interface. Enter:
sdmmtc
OR
cbmmtc
The system prompt changes to >. 3 Access the applications (APPL) level of the CS 2000 Core Manager or CBM maintenance interface and verify that the Bootp Loading Service and File Transfer Service applications are in service (.). Enter: appl Example system response:
Procedure steps
71
In SDM, the application "Bootpd and tftpd" is an optional application and may not be installed on your system. In this case it would not be visible in the APPL level menu. In CBM, "Bootpd and tftpd" does not have a presence on the maintenance interface APPL level. 4 If these applications are not in service, busy (Bsy) then return to services (RTS) the applications. For information about busying applications and returning them to service, see the Nortel CS 2000 Core Manager Administration and Security (NN10170-611) or Nortel Core and Billing Manager 800 Security and Administration (NN10362-611). If these applications are in service (.), check for bootpd and tftpd messages in the /var/adm/syslog and /var/adm/daemon.log. Unless log entries relating to application problems were generated, no log file exists for daemon.log.
At the SAM21 frame
5 6
Verify that the GWC card has power by looking for the lighted yellow or green LEDs on its faceplate. Use a VT100 terminal or a PC with terminal application software to connect to the DB9 serial port on the faceplate of the GWC card. Use a standard straight through serial cable, rather than a null modem cable.
7 8 9 10
Configure the PC software to set the PC serial port to 9600 baud, 8 bits, no parity, 1 stop bit. Start the terminal application and select a direct connection from COM1. Press and hold the reset button on the faceplate of the GWC card for 5 seconds. Monitor the boot process on the terminal.
If the boot fails, check for the error number and reference it to Table 8 "Possible boot errors" (page 72).
--End--
Job aid
Table 8 Possible boot errors Error ID 0500 Reason text TFTP retry time out. The following problems could exist:

0600
network has too much traffic the CS 2000 Core Manager or CBMis busy the tftp daemon is not running the load name was entered incorrectly
BOOTP retry time out. The following problems could exist:

8100
network has too much traffic the CS 2000 Core Manager or CBM is busy the bootp daemon is not running the /etc/bootptab file is incorrectly configured
The load file on the CS 2000 Core Manager or CBM has the wrong path, the wrong permissions, or the wrong load name. Message CRC errors. The network could be busy and causing traffic errors. 10baseT link failure. Verify that the Ethernet cable is fully seated in the faceplate and the router.
0020 0017
Procedure steps
73
Performing a GWC line data integrity audit

This procedure describes how to perform an on-demand data integrity audit of line data stored in the CS 2000 GWC Manager database (SESM), the Session Server (SS) Manager database (SIP lines), and the CS 2000 Core database. Use this procedure to check for defective data after line provisioning, or if you suspect there is a problem with line provisioning.
ATTENTION
Do not run the audit while line provisioning is occurring.
Prerequisites
Review and be familiar with concepts, restrictions and limitations related to a line data integrity audit. For information, see Data integrity audits (page 24). Ensure that there is no other line audit running or scheduled to run. Only one line audit can be in progress at any given time. An in-progress line audit blocks all attempts to run any subsequent line audit requests. If you run an on-demand line audit, and if that audit is still in progress at the start time of a scheduled audit, the scheduled audit will not occur. If you wish to view the currently running audits, see procedure Viewing or aborting running data integrity audits (page 107).
Procedure steps
Step 1 Action At the CS 2000 Management Tools window, select Maintenance, then Audit System.
The Audit System dialog box appears.
Click the Audit box and from the list of audits, select Line Data Integrity Audit.
3 4
Click the Run Audit button. At this time, the system performs the following checks to ensure that it is ready for the audit:
Are there 10 or more non-query line provisioning operations running? Is the batch provisioning tool (BPT) running a lines batch operation? Is there any bulk query operation in progress?
After each check, if the condition is found, the system displays an appropriate warning message. After each warning, you can click the Cancel button to postpone the audit and repeat this procedure at a later time. If you wish to continue, click the Proceed button on each displayed warning dialog box. If you click Proceed, the system displays Line Data Integrity audit selection window.
Procedure steps
75
The Granular Line Audit list on the left displays all the small and large line GWCs and the associated gateways, as well as the logical line groups (LGRP). You can display or hide any of the sub-lists by clicking on the plus (+) or minus (-) sign. 5 Read the following guidelines, then select the elements that you wish to audit.
Ensure that the CICM Audit box is not checked. If you wish to select more than one item at the same time, press the Shift key and click each item that you want to select. Do not select GWCs and LGRPs for the same audit. If you attempt to select both, the system displays an error message. If you try to select a GWC element and an LGRP is already selected (or the opposite), the system displays the following warning:
Click Replace to continue or click Close to start a new selection.
If you click the Integrity Audit check box (to select all GWCs and associated gateways or all logical groups), the system displays the following confirmation message:
Click Yes if you wish to continue. Otherwise, click No. Select the elements that you wish to audit. From the list on the left,
OR
click and highlight the specific item that you want to audit (single-direction audit)
click the Integrity Audit check box to select all GWCs and associated gateways or all logical groups (LGRP) (double-direction audit). If you check the Integrity Audit box, the system displays the "IsDNAudited?" panel.
If you want the system to audit directory-number (DN) data as part of the line audit, click Yes in the "IsDNAudited?" panel. If you click Yes in the "IsDNAudited" panel, the system displays the following panel.
If you want the system to audit directory-number (DN) data as part of the line audit, click Yes to continue. If you were to click No in the "IsDNAudited" panel, the system would display the following panel.
Procedure steps
77
6 7
Click the Add button. Your selection appears in the Selected box on the right. If necessary, repeat the selection process until all the items that you want to audit appear in the Selected window. If you need to remove an item from the Selected table, click the item to highlight it, then click the Rem button.
Click the Run Audit button to start the audit. During a line audit, the system displays a progress window, which includes the progress bar and the current operation.
The audit may take a few minutes to complete. When the audit completes successfully, the system displays the following message: Line Data Integrity Audit Completed. See report files. If the audit fails, the system displays the Line Data Integrity Audit Failed to Complete message with an error message indicating the reason. Contact your next level of support to resolve the problem.
10 11
Click the Close button to close the Audit Status window. To view the line audit reports, follow procedure Viewing GWC line data integrity audit reports (page 81).
--End--
Performing a CICM line data integrity audit

This procedure describes how to perform an on-demand data integrity audit of Centrex IP Client Manager (CICM) lines. Use this procedure to check for defective data after line provisioning, or if you suspect there is a problem with line provisioning.
ATTENTION
Do not run the audit while line provisioning is occurring.
Prerequisites
Make sure that the CICM Manager runs (I)SN09 MR6 or later load and CBM/SDM and Communication Server 2000 components run (I)SN10 or later loads. Review and be familiar with concepts, restrictions and limitations related to a line data integrity audit. For information, see Data integrity audits (page 24).
Procedure steps
The Audit System dialog box appears. 2 Click the Audit box and from the list of audits, select Line Data Integrity Audit.
Procedure steps
79
3 4
Click the Run Audit button. At this time, the system performs the following checks to ensure that it is ready for the audit:
Are there 10 or more non-query line provisioning operations running? Is the batch provisioning tool (BPT) running a lines batch operation? Is there any bulk query operation in progress?
After each check, if the condition is found, the system displays an appropriate warning message. After each warning, you can click the Cancel button to postpone the audit and repeat this procedure at a later time. If you wish to continue the audit, click the Proceed button on each displayed warning dialog box. If you click Proceed, the system displays Line Data Integrity audit selection window. 5 6 Select the CICM Audit check box. Select your audit options:
If you want to audit specific CICM nodes select the required GWC or LGRP, or both nodes from the list and ensure that the Select ALL box is not checked. If you want to audit all the CICM nodes check the Select ALL box. If you want to run the audit in Correction mode check the Correct CICM Datafill box. If you want to run the audit Report Only mode ensure that the Correct CICM Datafill box is not checked.
Click the Run Audit button to start the audit.

The system displays the appropriate confirmation dialog box listing the selected audit parameters. If the displayed audit parameters are not correct, click No and go back to the previous step to select the correct parameters. Otherwise, click Yes to start the audit. The system displays a progress window, which includes the progress bar and the current operation. 8 The audit may take a few minutes to complete. When the audit completes successfully, the system displays the following message: Line Data Integrity Audit Completed. See report files. If the audit fails, the system displays the Line Data Integrity Audit Failed to Complete message, as well as an error message indicating the reason. Contact your next level of support to resolve the problem. 9 10 Click the Close button to close the Audit Status window. To view the audit report, follow procedure Viewing GWC line data integrity audit reports (page 81).
--End--
Procedure steps
81
Viewing GWC line data integrity audit reports

Use this procedure to view the results of completed line data integrity audits stored on the CS 2000 Management Tools server, on the active unit of the GWC node. You can configure the number of reports that the CS 2000 Management Tools server retains. The maximum default value is seven. When a new line audit occurs, the server deletes the oldest report. The system places data audit reports in the following directory on the CS 2000 Management Tools server: /opt/nortel/ptm/current/www/Audit/Lin eDataIntegrityAudit/.
ATTENTION
Procedure steps
The Audit System dialog box appears. 2 Click the Audit box and from the list of audits, select Line Data Integrity Audit .
Click the Report box and from the drop-down menu, select the ValidLineData or the ProblemLineData report that you want to view. If there is more than one report, verify the and time information in the report names and decide which report you want to view.
Click the View Report button. The system displays the selected report. Example valid line data report
Procedure steps
83
Example problem line data report
If the audit found no problems, the "Problem" report contains a message stating that no problems were found. For Centrex IP Client Manager (CICM)-specific examples, see Job aid (page 84). 5 This step applies only to problem reports. Review the report to see if it contains messages in the following formats:
DN <dn> for LEN <len> has no associated endpoint on <GWC ID> To correct this type of problem:
Log in directly to the CS 2000 Core (XA-Core or Compact Call Agent). Access SERVORD and use the OUT command to remove the DN from the Call Agent. Delete the LEN from the Core table LNINV.
Endpoint <endpoint_name> on gateway <gateway_name> on <GWC_name> has no associated DN/LEN on the CM To correct this type of problem: Log in directly to the CS 2000 Core (XA-Core or Compact Call Agent). Manually add the LEN to table LNINV.
If you want to retain one of these reports for a longer time, or if you want to print a report, click the Save as button at the bottom of the screen to save the report under a new file name. To print the saved report, open the file using text editor and print the file. For a description of each problem, see the printed copy of the ProblemLineData report. To correct the problems, you need to delete and re-provision the listed lines. After viewing the report, click the Exit button at the bottom of the screen.
--End--
7 8
Job aid
This job aid contains examples of CICM line audit outputs.
Valid Line Data log

The valid line data log may contain lines that are also contained in the problem line data log. This occurs in the following circumstances:
The feature data is valid but the audit warns that a user is not assigned. The audit has found and corrected a problem, so the line is valid post audit.
The following figure shows an example of a valid line data log as viewed from the audit log viewer.
Job aid
85
The following table lists the fields shown in the valid line data log.
Table 9 Fields in a valid data log Field name LEN GW name GWC EP name CICM line User Enterprise Profile Description Line entry from LNINV table (KSETLINE) Name of CICM GW provisioned at CS 2000 Management Tools server Name of GWC hosting the CICM GW Name of CICM endpoint CICM line number corresponding to endpoint/LEN User assigned to line (may be empty) Enterprise associated with user (SN10+ CICM node) Profile that user is associated with (SN09 MR6+ CICM node)
Problem Line Data log

The problem line data log contains a set of zero or more problem report strings indicating problems that require correction, have been corrected or are flagged for attention, for example, a missing user.
The following figure shows an example of a problem line data log where the audit has not been allowed to auto correct the detected problems.
The following figure shows an example of a problem line data log where the audit has been allowed to auto correct the detected problems.
Job aid
87
The following table shows the problems that may be listed in the problem line data log.
Table 10 Possible problems in a line data log Problem HASU_USER Description Line in HASU state but has user entry at CICM. Example GWC = GWC-102 GW = CICM-333 EP = tp/0/0456 CICM Line = 0456. User ID 12345678@greenlan detected on HASU line. GWC = GWC-102 GW = CICM-333 EP = tp/1/0001 CICM Line = 1024. No user provisioned on line. GWC = GWC-102 GW = CICM-333 EP = tp/2/0121 CICM Line = 2167. Provisioned extension module quantity mismatched at CICM. GWC = GWC-102 GW = CICM-333 EP = tp/2/0002 CICM Line = 2049. Provisioned extension module unsupported by CICM. GWC = GWC-102 GW = CICM-333 EP = tp/2/0002 CICM Line = 2049. Provisioned line LCC type unsupported by CICM. GWC = GWC-102 GW = CICM-333 EP = tp/2/0122 CICM Line = 2168. Missing features on CICM key 5. GWC = GWC-102 GW = CICM-333 EP = tp/2/0122 CICM Line = 2168. Mismatched features on CICM key 12. GWC = GWC-102 GW = CICM-333 EP = tp/2/0122 CICM Line = 2168. Unexpected features on unassigned CICM key 3.
MISSING_USER
Line in WORKING state but no provisioned user.
EXT_QUANTITY
Mismatch between number of provisioned M522 modules.
ILLEGAL_EXT
Extension module type provisioned at call server not supported by CICM.
MISMATCH_LCC
LCC type provisioned a call server not supported by CICM.
FEAT_MISSING
Provisioned feature is not present on designated key.
FEAT_MISMATCH
Provisioned feature mismatch between CM and CICM.
UNEXPECTED_FEAT
No feature at CM but key provisioned at CICM.
Problem INCORRECT_DN
Description DN number at CICM does not match the CM.
Example GWC = GWC-102 GW = CICM-333 EP = tp/2/0122 CICM Line = 2168. Incorrect DN on CICM key 1. GWC = GWC-102 GW = CICM-333 EP = tp/0/0122 CICM Line = 0123. Incorrect User Label on CICM key 6. GWC = GWC-102 GW = CICM-333 EP = tp/2/0123 CICM Line = 2169. Multiple mismatches detected on line. GWC = GWC-102 GW = CICM-333 EP = tp/2/0124 CICM Line = 2170. Features detected on HASU line. GWC = GWC-102 GW = CICM-333 EP = tp/2/1000 CICM Line = 3046. Line not located in CICM database. GWC = GWC-102 GW = CICM-333 EP = tp/0/0456 CICM Line = 0456. User ID 11111111@redlan deleted from HASU line. GWC = GWC-102 GW = CICM-333 EP = tp/2/0121 CICM Line = 2167. Corrected extension module quantity mismatch. GWC = GWC-102 GW = CICM-333 EP = tp/2/0122 CICM Line = 2168. Added missing features to CICM key 5. GWC = GWC-102 GW = CICM-333 EP = tp/2/0122 CICM Line = 2168. Corrected mismatched features on CICM key 11. GWC = GWC-102 GW = CICM-333 EP = tp/2/0122 CICM Line = 2168. Deleted unexpected features on unassigned CICM key 2.
INCORRECT_LABEL
User string at CICM does not match that provisionable by flowthrough provisioning. Large number of mismatches at CICM; not displayed individually to prevent flooding log. Feature provisioned against a HASU line
MULTI_ERRORS
HASU_FEAT
MISSING_LINE
LEN entry at CM but corresponding CICM line is missing. HASU user at CICM deleted.
DEL_HASU_USER
CORRECT_EXT
M522 extension set mismatch corrected at CICM.
ADD_FEAT
Missing feature code added to key on CICM.
CORRECT_FEAT
Feature code mismatch corrected at CICM.
DEL_FEAT
Deleted feature code from CICM key.
Job aid
89
Problem CORRECT_DN
Description Corrected DN number on CICM key.
Example GWC = GWC-102 GW = CICM-333 EP = tp/2/0122 CICM line = 2168. Corrected DN on CICM key 1. GWC = GWC-102 GW = CICM-333 EP = tp/1/0005 CICM line = 1028. Corrected User Label on CICM key 7. GWC = GWC-102 GW = CICM-333 EP = tp/2/0123 CICM Line = 2169. Corrected multiple mismatches on line. GWC = GWC-102 GW = CICM-333 EP = tp/2/0124 CICM Line = 2170. Deleted features from HASU line. GWC = GWC-102 GW = CICM-333 EP = tp/2/0124 CICM Line = 2170. CICM XMLGWLineParser has detected an unsupported problem type. GWC = GWC-102 GW = CICM-333 EP = tp/2/0124 CICM Line = 2170. "Corrupt user detected on line. Disassociate from line. GWC = GWC-102 GW = CICM-333 EP = tp/2/0124 CICM Line = 2170. Corrupt user removed from line. GWC = GWC-102 GW = CICM-130 EP = tp/0/0000 CICM Line = ????. Unable to audit CICM Line. Node Missing from CICM or communication problem. Unsupported problem report: 35
CORRECT_LABEL
Corrected label mismatch on CICM key.
MULTI_CORRECT
Large number of corrections performed against CICM line. Not listed individually to prevent flooding log. All line data fill removed from a HASU line at CICM.
DEL_HASU_LINE
UNEXPECTED_REPORT
Problem report type not support by CICM EM. Not end user visible.
CORRUPT_USER
Corruption of user data fill detected at CICM.
CORRECT_CORRUPT_ USER
Corruption of user data fill detected at CICM and corrected by audit. CICM is unable to audit line - indicates Node is out of service or incorrect provisioning. This version of the CS 2000 Management Tools server does not support the received type of problem report.
AUDIT_LINE_NOT_ POSSIBLE
UNSUPPORT_REPORT
Performing a GWC trunk data integrity audit

This procedure describes how to perform an on-demand data integrity audit of trunk data stored in the CS 2000 GWC Manager database and the CS 2000 Core database. Use this procedure to check for defective data after trunk provisioning, or if you suspect there is a problem with trunk provisioning.
ATTENTION
Do not run the audit while trunk provisioning is occurring.
Prerequisites
Review and be familiar with concepts, restrictions and limitations related to a trunk data integrity audit. For information, see Data integrity audits (page 24). Ensure that there is no other trunk audit running or scheduled to run. Only one trunk audit can be in progress at any given time. An in-progress trunk audit blocks all attempts to run any subsequent trunk audit requests. If you run an on-demand trunk audit, and if that audit is still in progress at the start time of a scheduled audit, the scheduled audit will not occur. If you wish to view the currently running audits, see procedure Viewing or aborting running data integrity audits (page 107).
Procedure steps
Procedure steps
91
Click the Audit box and from the list of audits, select Trunk Data Integrity Audit.
Click the Run Audit button. During the audit, the system displays a progress window, which includes the progress bar and the current operation.
The audit may take a few minutes to complete. When the audit ends successfully, the system displays the following message: Trunk Data Integrity Audit Completed. Please see the report file. If the audit fails, the system displays the Trunk Data Integrity Audit Failed to Complete message with an error message indicating the reason. Contact your next level of support to resolve the problem.
5 6
Click the Close button to close the Audit Status window. To view trunk audit reports, follow procedure Viewing GWC trunk data integrity audit reports (page 92).
--End--
Viewing GWC trunk data integrity audit reports

Use this procedure to view the results of completed trunk data integrity audits stored on the CS 2000 Management Tools server. The CS 2000 Management Tools server retains the six most recent reports. When a new trunk audit occurs, the server deletes the oldest report. The system places trunk audit reports in the following directory on the CS 2000 Management Tools server: /opt/nortel/ptm/current/www/Audit/ TrunkDataIntegrityAudit/.
ATTENTION
Procedure steps
The Audit System dialog box appears. 2 Click the Audit box and from the list of audits, select Trunk Data Integrity Audit .
Procedure steps
93
Click the Report box and from the drop-down menu, select the ValidTrunkData or the ProblemTrunkData report that you want to view. If there is more than one report, verify the date and time information in the report names and decide which report you want to view.
Click the View Report button. The system displays the selected report. Example of a ValidTrunkData report
Example of a ProblemTrunkData report If the audit found no problems, the report contains a message stating that no problems were found. The ProblemTrunkAudit report can contain messages in the following formats:
Trunk <trunk name> (node number = <NODE>, terminal number = <TID>) has no associated endpoint on GWC <GWC ID>. Endpoint <EP NAME> on gateway <GW NAME> (terminal number = <TID>) on GWC <GWC ID> has no associated trunk member datafilled on the CM.
Procedure steps
95
If you want to retain one of these reports for a longer time, or if you want to print a report, click the Save as button at the bottom of the screen and save the report under a new file name. To print the saved report, open the file using text editor and print the file. See the printed copy of the ProblemTrunkData report for a description of each problem. To correct the problems, delete and re-provision the listed trunks. After viewing the report, click the Exit button at the bottom of the screen.
--End--
6 7
Performing a CS 2000 data integrity audit

This procedure describes how to perform an on-demand data integrity audit of the CS 2000 Gateway Controller (GWC) Manager, CS2000 Core, and Session Server Manager databases. Use this procedure when you are receiving logs or alarms on the CS 2000 Core or at the CS 2000 SAM21 Manager client indicating a possible provisioning errors or data inconsistencies. When the audit is running, suitable locks are in place that disable provisioning operations.
Prerequisites
Review and be familiar with concepts, restrictions and limitations related to a CS 2000 data integrity audit. For information, see Data integrity audits (page 24). Ensure that no provisioning activities are scheduled to take place during the audit. Ensure that there is no other line audit running or scheduled to run. Only one line audit can be in progress at any given time. An in-progress line audit blocks all attempts to run any subsequent line audit requests. If you run an on-demand line audit, and if that audit is still in progress at the start time of a scheduled audit, the scheduled audit will not occur.
Procedure steps
Procedure steps
97
Click the Audit box and from the list of audits, select CS2K Data Integrity Audit.
At the Audit System dialog box, click the Run Audit button to display the CS2K Data Integrity Audit Configuration dialog box.
Select the CS 2000 component that you wish to audit:
Select the CS2KSS EM Data Integrity Audit check box if you wish to verify that the network zones configuration
data in the GWC Manager and the Session Server Manager databases match.
Select the CS2K Call Server Data Integrity Audit check box if you wish to verify that the data held by the GWC Manager database and the CS 2000 Call Server Core match. Select both check boxes if you wish to audit both components.
If there is no Session Server Manager configured on the CS 2000, the CS2KSS EM Data Integrity Audit check box is disabled. 5 Click the Run Audit button to start the audit. If you wish to cancel the process, click the Close button. During a CS 2000 audit, the system displays a progress window with a progress bar. 6 The audit may take a few minutes to complete. When the audit ends successfully, the system displays the following message: Audit Completed. Please see report files. If the audit fails, the system displays the CS2K Data Integrity Audit Failed to Complete message, with an error message indicating the reason. Contact your next level of support to resolve the problem. 7 8 Click the Close button to close the Audit Status window. To view the audit report, follow procedure Viewing CS 2000 data integrity audit reports (page 99).
--End--
Procedure steps
99
Viewing CS 2000 data integrity audit reports

Use this procedure to view the results of completed CS 2000 data integrity audits. The system places the audit report in the following directory on the CS 2000 Management Tools server: /opt/nortel/ptm/current/MI2/apps/Audit. The CS 2000 Management Tools server retains the most recent CS 2000 audit report. When a new audit occurs, the server deletes the previous report. The CS 2000 GWC Manager does not provide an option to save a CS 2000 data audit report to the local disk.
ATTENTION
Procedure steps
The Audit System dialog box appears. 2 Click the Audit box and from the list of audits, select CS2K Data Integrity Audit .
100
Click the Report box and from the drop-down menu, select the report that you want to view. The file name has the following format: Report-<date> Click the View Report button. The system displays the selected report. If no problems were discovered, the report is empty. Example of a CS2K audit report
Procedure steps
101
The CS2KSS EM Data Integrity Audit attempts to automatically correct all data mismatches found. The problems listed in the report are typically corrected and are displayed for information only. If any mismatch correction fails, manual action may be required or the system reattempts the audit, or both. 5 Review the results of the audit and select the problem to resolve. If necessary, re-size the entire window to view the complete Problem Description field. 6 7 8 Click the Actions box and from the drop-down menu, select the appropriate action. Read the description of the action and ensure that you observe any recommended steps or cautions. Click the Take Action button. If you see the message Correction Failed, contact your next level of support. 9 Return to Step 5 if you wish to review another problem.
--End--
102
Performing a GWC V5.2 data integrity audit

This procedure describes how to perform an on-demand data integrity audit of V5.2 interfaces. Use this procedure to check for defective data after you have done V5.2 interface provisioning, or if you suspect there is a problem with V5.2 provisioning. When the audit is running, suitable locks are in place that disable provisioning operations.
Prerequisites
Review and be familiar with concepts, restrictions and limitations related to a V5.2 data integrity audit. For information, see Data integrity audits (page 24), Ensure that no provisioning activities are scheduled to take place during the audit. Ensure that there is no other V5.2 audit running or scheduled to run. Only one V5.2 audit can be in progress at any given time. An in-progress audit blocks all attempts to run any subsequent audit requests. If you run an on-demand V5.2 audit, and if that audit is still in progress at the start time of a scheduled audit, the scheduled audit will not occur. If you wish to view the currently running audits, see procedure Viewing or aborting running data integrity audits (page 107).
Procedure steps
Step 1 Action At the CS 2000 Management Tools window, from the Maintenance menu, select Audit System.

Procedure steps
103
Click the Audit box and from the list of audits, select V5.2 Data Integrity Audit .
Click the Run Audit button to start the audit. During the audit, the system displays the Audit Status message box.
The audit may take a few minutes to complete. When the audit completes successfully, the system displays one of two types of messages :
V5.2 Data Integrity Audit Completed. No Problems Found. V5.2 Data Integrity Audit Completed. 2 Problems Found.
If the audit fails, the system displays the V5.2 Data Integrity Audit Failed to Complete message with an error message indicating the reason. Contact your next level of support to resolve the problem. 5 6 Click the Close button to close the Audit Status box. To view V5.2 audit reports, complete procedure Viewing GWC V5.2 data integrity audit reports (page 104).
--End--
104
Viewing GWC V5.2 data integrity audit reports

Use this procedure to view the results of completed V5.2 interface audits. The CS 2000 Management Tools server retains the most recent report. When a new audit occurs, the server deletes the previous report. The system places the audit report in the following directory on the CS 2000 Management Tools server: /opt/nortel/ptm/current/MI2/apps/Audit. The CS 2000 GWC Manager does not provide the option to save a V5.2 audit report to the local disk.
ATTENTION
Procedure steps
Step 1 Action At the CS 2000 Management Tools window, from the Maintenance menu, select Audit System .
The Audit System dialog box appears. 2 Click the Audit box and from the list of audits, select V5.2 Data Integrity Audit.
Procedure steps
105
Click the Report box and from the drop-down menu, select the report that you want to view. The file name has the following format: Report-<date> Click the View Report button. The system displays the selected report. If the audit does not discover any problems, the report is empty.
Review the results of the audit and select a problem to resolve. If necessary, re-size the entire window to completely view the Problem Description field.
106
6 7
Click and hold on the Action drop-down menu near the bottom of the screen to assess any possible actions. If appropriate, select an action. Read the description of the action and ensure that you observe any recommended steps or cautions. Click the Take Action button. If you see the message Correction Failed, contact your next level of support.
If required, return to Step 5 to review another problem.

--End--
Procedure steps
107
Viewing or aborting running data integrity audits

Use this procedure to view and, if required, abort currently running audits. You can view and, if necessary, abort any of the following audits:
line audit (including CICM line audit) trunk audit V5.2 audit CS 2000 (CS2K) data integrity audits:
CS2K Call Server data integrity audit CS 2000 Session Server (CS2KSS) Manager data integrity audit
Procedure steps
Step 1 Action At the CS 2000 Management Tools window, click the Maintenance menu and select View Current Audits.
The Audit View window appears, displaying currently running audits. 2 If you wish to abort one of the running audits, click the appropriate row to select the audit. Your selection is highlighted.
108
Click the Abort button. At the confirmation window, click OK to continue. The audit originator receives an error message indicating that the audit was aborted and identifying the user that aborted the audit.
--End--
Procedure steps
109
Setting up protocol trace environment on the CS 2000 Management Tools server

This procedure describes how to set up the trace environment on the CS 2000 Management Tools server, before using the Protocol Trace tool on the Gateway Controller (GWC) to trace protocol packets. Use this procedure only if you are sent here from procedure Tracing GWC protocol packets (page 112).
Procedure steps
Step 1 2 3 4 5 Action Telnet to the CS 2000 Management Tools server: telnet <server> When prompted, enter your user ID and password. The system prompt changes to a dollar sign ($) Change to the root user: su -root When prompted, enter the root password. Create a directory where the GWCs place the capture (.cap) files. Enter: mkdir <new_directory_name> 6 Add a new user ID to the mgcmtc user group: useradd -g mgcmtc -d /<directory_name> <new_userid> You will use this ID to telnet to the GWC and to send (ftp) .cap files to the CS 2000 Management Tools server. 7 8 9 Assign the newly created user ID to the succession user group: usermod -g succssn -G mgcmtc <userid> Create a password for the user ID you just created: passwd -r files <userid> When prompted, enter a password of at least three characters. For more information about how to add new users to user groups, see procedure "Setting up local user accounts on an SPFS-based server" in Nortel ATM/IP Solution-level Administration and Security ( (NN10402-600)).
At your workstation
110
10 11
Associate directory ownership with the created user. Enter: chown <userid>:mgcmtc /<directory_name> Set up the FTPD user on the CS 2000 Management Tools server. Follow procedure "Adding a user to Proftpd" in Nortel ATM/IP Solution-level Administration and Security ( (NN10402-600)) using the following configuration values:

12
default user path: /<directory_name> user_id: <userid> group_id: mgcmtc maximum clients for this user: 10 password required for <userid> in group mgcmtc: y Enter the same password you created for the new user in Step 9. default path read only: n default path write only: n user allowed to change directory: n user allowed to create/delete directory: n user allowed to delete files: y user allowed to rename files: y user allowed to overwrite files: y
Verify the amount of free disk space on the CS 2000 Management Tools server. Enter: df -k If required, more space can be allocated, but Nortel does not recommend it.
13
Set a limit file space for the FTPD user configured in step Step 11. Follow procedure "Setting disk quota for FTP users" in Nortel ATM/IP Solution-level Administration and Security ( (NN10402-600)). Provided that the server has enough disk space available, Nortel recommends that you set the disk quota to 100 000 Kbytes. This step ensures that the trace files sent from the GWC do not overrun the available disk space on the CS 2000 Management Tools server.
14
Verify that the disk quota is set. Use the following command outside of the CLI:
repquota -v /
111
15
Return to procedure Tracing GWC protocol packets (page 112).

--End--
Variable denitions
Variable <directory_name> <new_directory_name> <new_userid> <server> <userid> Value The name of the directory that you created. Example: PkTrace The name of the new directory. Example: PkTrace The name of the new user. Example: gwcUser The IP address or host name of the CS 2000 Management Tools server. The name of the user that you added. Example: gwcUser
112
Tracing GWC protocol packets

This procedure describes how to set up and use the Protocol Trace tool on the Gateway Controller (GWC). This tool allows you to set up a sniffer trace on the GWC and direct the output to the CS 2000 Management Tools server. Use this procedure as part of troubleshooting and fault clearing activities.
ATTENTION
Within a GWC telnet session, you can configure and start only one trace for each GWC unit.
Prerequisites
Before starting this procedure for the first time after an upgrade, you must set up the trace environment on the CS 2000 Management Tools server. Complete procedure Setting up protocol trace environment on the CS 2000 Management Tools server (page 109). Review information contained in section Job aid (page 118).
Procedure steps
Step 1 Action Connect to the GWC node on which you want to start the protocol trace. Enter: telnet <GWC_IP> 2 3 When prompted, login to the GWC. Enter: <userid> When prompted, enter your password. The system prompt changes to PROtocoltrace>. System response:
Warning! The trace captured may contain data that is restricted by Privacy or CALEA/LI Regulations. The user is responsible to ensure that the captured data storage, transmission, handling and disposal is within procedural, regulatory and legal bounds for jurisdiction(s) where the call is occurring.
At your workstation
Procedure steps
113
PROtocoltrace> Duration, END, Help, Ip, Query, Reset, Start, Tcp, Udp, View.
If you wish to list all available commands with a short explanation, use the Help command. Enter: h You can invoke this command at any time. System response:
You can execute each command (except End) by entering the full name of the command (for example, reset) or just the first letter (for example, r). 5 This step is optional. If you wish to obtain port numbers and external IP addresses for the gateways, complete one or more of the following actions (as applicable):
For all gateways provisioned through the GWC Manager GUI, use the GUI to obtain the port number and the IP address for each gateway. Follow procedure "Viewing gateway provisioning data for a GWC node" in Nortel Gateway Controller Configuration ( (NN10205-511)). If the GWC Manager GUI shows an IP address of 0.0.0.0 (a gateway is provisioned for a Domain Name Server [DNS] lookup), use the DNS to obtain the gateway IP address. For all gateways provisioned through the Core and for any non-gateway traces (such as, USP M3UA data), trace all packets for the GWC for 1 minute. Complete procedure Tracing all packets for a GWC (page 121), then continue with this procedure.
114
Configure trace filters using the following commands: IP and/or UDP, TCP. You can invoke these commands only when the trace is disabled. Otherwise, the system displays the following error message:
Command rejected: FTP in progress!
For the description of these commands, see section Trace filters (page 119).
ATTENTION
If you select an IP address with IPSec active, and
you do not specify any ports (trace filtering based on IP address only), the .cap file opened with Wireshark shows Encapsulated Security Payload (ESP) instead of the protocol name and the payload is not visible. you specify the port number (trace filtering based on IP address and port number), the .cap file opened with Wireshark shows the protocol name and the payload is visible.
Packets captured with UDP ports that use IPSec show an invalid IP checksum (checksum of 0), when .cap file is opened in Wireshark. This is not an error but an indication that the packet was processed to remove the IPSec content. For these packets, ignore the Ethernet addresses as well.
If you enter:
IP At prompts, enter the appropriate IP addresses. If the entered address is not correct, the system displays the following error message:
Invalid IP address
Example: PROtocoltrace> i Enter external IP Enter external IP Enter external IP Enter external IP
1 (0 2 (0 3 (0 4 (0
= no = no = no = no
IP): IP): IP): IP):
47.19.7.9 10.1.1.1 20.2.2.2 30.3.3.3
UDP At prompts, enter the appropriate port numbers within the range of 1 to 65535. If you enter an invalid number, the system displays the following error message:
Invalid port number (range 1 - 65535)
Procedure steps
115
Example: PROtocoltrace> u Enter UDP/SCTP Port 1 Enter UDP/SCTP Port 2 Enter UDP/SCTP Port 3 Enter UDP/SCTP Port 4
(0 (0 (0 (0
= N/A): 53 = N/A): 161 = N/A): 162 = N/A): 123
TCP At prompts, enter the appropriate port numbers within the range of 1 to 65535. If you enter an invalid number, the system displays the following error message:
Invalid port number (range 1 - 65535)
Example: PROtocoltrace> t Enter TCP Port 1 (0 Enter TCP Port 2 (0 Enter TCP Port 3 (0 Enter TCP Port 4 (0 7
= N/A): 23 = N/A): 21 = N/A): 1720 = N/A): 0
Configure the duration of the trace using the Duration command. The default value is 30 minutes. If you wish to change it, enter: d System response:
Time duration for trace in minutes (1-120):
Enter a new value (between 1 and 120 minutes). If you enter 0 or more than 120, the system displays the following error message:
Invalid Time duration (maximum 2 hours)
You can invoke the Duration command only when the trace tool is disabled. Otherwise, the system displays the following error message:
Verify your configuration using the Query command. Enter: q Example response:
TCP ports: 23 21 1720 0 Duration set to 15 minutes Buffer wrap mode for buffer size 65536 Trace capture stopped
You can invoke the Query command at any time.
116
If trace filters are not configured, the response to the Query command is: Configure UDP, TCP and/or IP for trace! 10 If you wish to change the current configuration, use the Reset command to set all configuration values to default. Then, configure the new trace filters and new time interval as described in previous steps. You can also change any of these configuration values separately at any time by invoking the appropriate command and entering the new value. For example, if you only wish to change the UDP port, enter the UDP command and enter the new port value. If the displayed current configuration is correct, continue with the next step. 11
CAUTION
Possible loss of data
If there any previously captured files from this GWC unit in the FTPD directory on the SESM destination, issuing the Start command will erase those files! If their data needs to be saved prior to starting a new trace, retrieve them first as described in procedure Retrieving protocol trace files (page 123).
Only one trace session may be in progress at any time for each GWC. Otherwise, the system displays the following error message:
Start the trace. Enter: s System response:

Enter FTP User name (max 32 char): Enter FTP Password (max 32 char):
12
At the prompt, enter the user ID and password that you created in procedure Setting up protocol trace environment on the CS 2000 Management Tools server (page 109). The password that you enter is displayed on the telnet session screen.
13
If you wish to view the trace in progress, use the View command. Enter: v
Procedure steps
117
You can invoke the View command at any time, but if you enter this command before the trace starts, the system responds with the following message:
Trace stopped with reason: not set
Example response, when the trace is in progress:

Trace capture in progress, FTP in progress to host: <CS 2000 Management Tools server IP address> Buffered message count = <number of messages in the buffer> Buffered bytes received = <number of bytes in the buffer> Total message count = <total number of messages captured>
14
Wait for the trace to complete (based on the configured duration) or terminate the trace using the End command. To avoid an accidental entry of this command, you must enter the full name of the command. If you use the End command, all captured data will be sent to the CS 2000 Management Tools server, then the last file will close. To stop the trace, enter: end Once all the data is successfully sent to the CS 2000 Management Tools server, the system displays the following message:
Continuous Ethereal Trace is done!
If a problem occurs during the trace, the system aborts the process and displays the following message:
Continuous Ethereal Trace is aborted!
15
To display the trace details, enter: v Example response when the trace ends after the time duration expires:
Trace stopped with reason: trace done Buffer message count = <number of messages in the buffer> Buffered bytes received = <number of bytes in the buffer> Total message count = <total number of messages captured>
In this scenario, the total message count matches the number of packets in the CAP file. Example response when you use the End command to stop the trace:
118
Fault monitoring and troubleshooting procedures Trace stopped with reason: user end cmd Buffer message count = <number> Buffered bytes received = <number> Total message count = <number>
For a list of reasons displayed when the trace is aborted, see job aid Reasons for aborting the trace (page 120). 16 To set all configuration values back to default, use the command Reset. Enter: r All ports and IP addresses are set to 0 (no filtering) and the time duration is set to 30 minutes. You can invoke this command only when the trace is disabled. Otherwise, the system displays the following error message:
17
To retrieve the CAP file from the CS 2000 Management Tools server, complete procedure Retrieving protocol trace files (page 123). On you workstation, open the file with Ethereal or Wireshark and the appropriate Plug-in.
--End--
18
Variable denitions
Variable <GWC_IP> <userid> Value The GWC active IP address. The user ID that you created when setting up protocol trace environment on the CS 2000 Management Tools server (procedure Setting up protocol trace environment on the CS 2000 Management Tools server (page 109)).
Job aid
Protocols and port numbers
The following table lists protocols and most commonly used port numbers.
Table 11 Protocols and common port numbers Protocol BCP (Media Portal) COPS (DQos) In/Out port to capture 3904 2126 (TCP)
Job aid
119
Table 11 Protocols and common port numbers (contd.) Protocol DNS GCP GWC Connection Broker GWCCM/FCM H.248 / MEGACO H.323 (Q.931)/RAS(H.225) IUAP (Q.931) MGCP / NCS M3UA over SCTP M2UA over SCTP SNMP TGCP IKE V5.2 In/Out port to capture 53 7060, 7061 - SSL, 5060, 5061 - SST 22532 4684 2944 1719 UDP 1720 TCP 9900 2427 2905 2904 161 2427 500 10001
Trace lters
The following table describes the commands that you use to configure trace filters.
Table 12 Trace filter commands Command IP Description The IP command allows you to select up to four IP addresses for filtering against source or destination packets. If a TCP or UDP port is configured, the IP check is applied only to packets that passed the port filtering. If you do not specify any ports, the IP filtering is applied to all IP packets. TCP port 20 is used for FTP data transfer, so packets sent to this port are not captured, even if the IP address for the packet matches the configured IP address.
120
Command UDP
Description The UDP command allows you to select up to four UDP (user datagram protocol) or SCTP (stream control transmission protocol) ports for filtering against source or destination port. You can also use the IP command to provide additional filtering on source and destination IP address. The TCP command allows you to select up to four TCP (transmission control protocol) ports for filtering against source or destination port. TCP port 20 is used for FTP data transfer, so do not trace this port. If you attempt to trace port 20, the system displays the following error message: TCP port 20 not allowed. You can also use the IP command to provide additional filtering on source and destination IP address.
TCP
Reasons for aborting the trace

The following table lists the reasons displayed when the system aborts the trace.
Table 13 Reasons for aborting the trace Displayed reason ftp error or message loss ftp throughput Explanation Problem with FTP. If a file is created, it might be corrupted. Input capture had to stop to avoid buffer overlay of un-sent data. Trace buffer is too small to buffer captured data. The trace file contains packets captured until the error occurred. Imaging or patching forced the trace FTP to end. The trace file contains packets captured until the trace was forced to end.
ftp forced out
Action
121
Tracing all packets for a GWC

This procedure describes how to trace all packets for a GWC to obtain port numbers and external IP addresses of associated gateways. Use this procedure only if you are sent here from procedure Tracing GWC protocol packets (page 112).
Action
Step 1 2 Action Clear any previous trace settings. Enter:
reset
At the PROtocoltrace> prompt
Configure the IP trace filter. Enter:

ip
System response:
Enter external IP 1 (0 = no IP):
At the prompt, enter the GWCs active IP address. System response:

Enter external IP 2 (0 = no IP):
4 5
At the prompt, enter 0 (zero). Verify the setting. Enter:

q
Example response:
IP address: xxx.xx.xx.x Duration set to 30 minutes Buffer wrap mode for buffer size 65536 Trace capture stopped
Start the trace. Enter:

start
System response:
Enter FTP User name (max 32 char): Enter FTP Password (max 32 char):
At the prompt, enter the user ID and password that you created in procedure Setting up protocol trace environment on the CS 2000 Management Tools server (page 109).
122
The system displays various progress information, including the .cap file name sent to the CS 2000 Management Tools server. 8 After few minutes, stop the trace. Enter:
end
Once the transfer is complete, the system displays the following message:
Continuous Ethereal Trace is done!
Retrieve the trace file sent to the CS 2000 Management Tools server. Complete procedure Retrieving protocol trace files (page 123), then continue with the next step. On you workstation, open the file with Ethereal or Wireshark. From the displayed file, note the port numbers and external IP addresses for the gateways. Exit the current location. Enter:
logout
10 11 12 13
Return to the procedure that sent you here.

--End--
Action
123
Retrieving protocol trace les

Use this procedure to retrieve the Gateway Controller (GWC) protocol trace files from the CS 2000 Management Tools server.
Action
Step 1 2 Action Access the CS 2000 Management Tools server. Enter:
<ip_address>
At the ftp> prompt
When prompted, enter the user ID that you created when setting up the trace environment on the CS 2000 Management Tools server (procedure Setting up protocol trace environment on the CS 2000 Management Tools server (page 109)). When prompted, enter password. View the list of files. Enter:
ls
3 4
The system displays all trace files in the following format:

GWC-x-UNIT-y_n.cap
x is the GWC node number y is the unit number: 0 or 1 n is the file number, starting with 1
Example response:
Opening ASCII mode data connection for file list -rw-r-r-- 1 gwcpkt mgcmtc 477103 Jul 32 13:15 GWC-11-UNIT-1_1.cap Transfer complete.
5 6 7 8 9
Note the name of all trace files (with the extension .cap); for example, GWC-11-UNIT-1_1.cap. Change to binary mode. Enter:
bin
Retrieve all .cap files. Enter:

get <trace_cap_file>
Wait until the system displays the Transfer complete message. Repeat the two previous steps for all .cap trace files that you noted.
124
10
To free disk space for the next trace, delete all trace files within FTP from the CS 2000 Management Tools server in the descending order, starting with the last one created (with the highest number). For example, if there are three .cap files: _1.cap, _2.cap, _3.cap, delete the _3.cap first, then _2.cap, then _1.cap. Whenever a new trace starts, the GWC checks if a trace file exists and tries to delete it. If more than one file exist, the GWC only checks file number _1. If you delete file _1 first and the other files still exist, the GWC does not look for them and starts the trace. To delete a .cap trace file, enter:
del <transferred_trace_cap_file>
11 12
Exit the CS 2000 Management Tools server:

bye
Return to the procedure that sent you here.

--End--
Variable denitions
Variable <ip_address> <trace_cap_file> <transferred_trace_cap_file> Value The IP address of the CS 2000 Management Tools server. The name of the trace file (with the extension .cap); for example, GWC-11-UNIT-1_1.cap. The name of the transferred trace file; for example, GWC-11-UNIT-1_1.cap
125

This chapter contains GWC fault clearing procedures.
126
Clearing the GWC318 critical alarm manually

This procedure describes how to manually clear the GWC318 critical alarm. GWC318 critical alarm is raised when a GWC initiates an autonomous switch of activity (SWACT) in an attempt to restore communication with its associated media gateways. Use this procedure after the autonomous SWACT occurred and the problem on the original active unit is solved and you wish to clear GWC318 critical alarm manually.
Prerequisites
The autonomous SWACT occurred and the GWC318 critical alarm is raised. The problem on the original active unit is solved.
Procedure steps
Step 1 2 3 Action At the CS 2000 Management Tools window, click the Gateway Controller folder from the Device Types menu. From the Contents of: Gateway Controller frame, select the GWC node on which the alarm is raised. Click the Provisioning tab and the Controller tab.
Procedure steps
127
4 5 6
Click the GWC Autonomous Swact: Change button. The Change GWC Autonomous Swact dialog box opens. Click the Clear Alarm button. At the confirmation window, click OK to confirm your request. If you wish to cancel the operation, click No and then Cancel .
--End--
128
Clearing the GWC320 Phase 1 SA failure alarm

Use this procedure to clear a GWC320 critical alarm raised for the followin g specific problem: Phase 1 SA failure - <remote_gateway_IP_address>.
Prerequisites
Verify whether any security logs associated with the alarm exist in the syslog file. Follow procedure Viewing GWC logs in syslog files (page 56), searching for the text string ISAKMP_FAIL. An ISAKMP_FAIL log with the same date and time as the alarm may indicate the precise reason for the failure. For the description of all IKE/IPSec security logs, see section IPSec and IKE security logs (page 170)
Make sure that the remote media gateway is enabled and that the IPSec on the remote gateway is enabled.
Procedure steps
Step 1 Action Launch the CS 2000 Management Tools client application. If required, see procedure "Launching CS 2000 Management Tools and NPM client applications" in Nortel ATM/IP Solution-level Administration and Security ( (NN10402-600)). At the CS 2000 Management Tools window, click the Gateway Controller folder from the Device Types menu. From the Contents of: Gateway Controller frame, select the GWC node on which the alarm is raised. Click the Provisioning tab, the IPSec tab, then the Connection Policy tab to display connection policies currently configured for the selected GWC node.
At your workstation
2 3 4
In the Remote host address column, find the IP address that matches the <remote_gateway_IP_address> in the alarm.
Procedure steps
129
If there is more than one policy with the same Remote host address, select the one with the lowest policy ID. Click that row to highlight the policy. The policy with the lowest ID number has the highest priority and the GWC will select that policy. 6 If the highlighted policy is
BYPASS or DISCARD stop the procedure and contact your next level of support! FLEX or SECURE continue the procedure.
For the highlighted policy, verify that the IKE Authentication IP address matches the Remote host address. If these two addresses do not match, stop the procedure and reconfigure the connection policy correctly. For full instructions, see Nortel CVoIP IPSec Service Configuration ( (NN10453-100)). If these two addresses match, continue the procedure.
Click the IKE Authentication tab.
In the IKE remote gateway column, find the IP address that matches the <remote_gateway_IP_address> in the alarm. From the selected IKE authentication row, note the name of the IKE preference list. Click the IKE Preference List tab. Find the preference list name that matches the name noted in Step 9.
10
11
Note the names of all preferences included in the list (displayed in the Preference list column). Each preference list can include up to three preferences. Click the IKE Preference tab.
12
130
Under IKE preference name heading, find the preferences noted in the previous step and verify that the configuration values for at least one preference match the values configured on the remote gateway. If the remote gateway is a Media Gateway 9000, verify that one of the preferences has the following values:

13
Cipher algorithm = 3DES-CBC Hash algorithm = SHA Lifetime = 86400 Diffie-Hellman key group = 1
If the IKE preference values match, stop the procedure and contact your next level of support. If the IKE preference values do not match, go to the next step. Add a new IKE preference and IKE preference list with the configuration values that match the values configured on the remote gateway. If required, see procedure "Configuring IKE Preference and Preference List" in Nortel CVoIP IPSec Service Configuration ( (NN10453-100)).
14
15
Modify the IKE authentication selected in Step 9 by selecting the new IKE preference list. If required, see procedure "Modifying IKE authentication_change IKE preference list" in Nortel CVoIP IPSec Service Configuration ( (NN10453-100)).
16
If the alarm clears within 5 minutes, the procedure is complete. If the alarm does not clear within 5 minutes, contact your next level of support.
--End--
Procedure steps
131
Clearing the GWC320 Phase 2 SA failure alarm

Use this procedure to clear a GWC320 critical alarm raised for the followin g specific problem: Phase 2 SA failure - <remote_gateway_IP_address>.
Prerequisites
Verify whether any security logs associated with the alarm exist in the syslog file. Follow procedure Viewing GWC logs in syslog files (page 56), searching for the text string ISAKMP_FAIL. An ISAKMP_FAIL log with the same date and time as the alarm may indicate the precise reason for the failure. For the description of all IKE/IPSec security logs, see procedure IPSec and IKE security logs (page 170).
Make sure that the remote media gateway is enabled and that the IPSec on the remote gateway is enabled.
Procedure steps
Step 1 Action Launch the CS 2000 Management Tools client application. If required, see procedure "Launching CS 2000 Management Tools and NPM client applications" in Nortel ATM/IP Solution-level Administration and Security ( (NN10402-600)). At the CS 2000 Management Tools window, click the Gateway Controller folder from the Device Types menu. From the Contents of: Gateway Controller frame, select the GWC node on which the alarm is raised. Click the Provisioning, IPSec , then Connection Policy tabs to display connection policies currently configured on the selected GWC node.
At your workstation
2 3 4
In the Remote host address column, find the IP address that matches the <remote_gateway_IP_address> in the alarm.
132
If there is more than one policy with the same Remote host address, select the one with the lowest policy ID. Click that row to highlight the policy. The policy with the lowest ID number has the highest priority and the GWC will select that policy. The remote host IP address must match the IKE Authentication gateway IP address displayed in the same row. 6 If the highlighted policy is

7 8
BYPASS or DISCARD stop the procedure and contact your next level of support. FLEX or SECURE continue the procedure.
Note the name of the IPSec preference list configured for the selected connection policy. Click the IPSec Preference List tab. Find the preference list name that matches the name noted in the previous step.
Note the names of all preferences included in the list (displayed in the Preference list column). Each preference list can include up to five preferences. Click the IPSec Preference tab.
10
11
Under IPSec preference name heading, find the preferences noted in Step 9 and verify that the configuration values for at least one preference match the values configured on the remote gateway. If the remote gateway is a Media Gateway 9000, verify that one of the preferences has the following values:
ESP cipher algorithm = ESP_NULL ESP HMAC algorithm = SHA
Procedure steps
133

12
Lifetime = 57600 PFS: Diffie-Hellman key group = 1
If IPSec preference values

13
match stop the procedure and contact your next level of support. do not match go to the next step.
Add a new IPSec preference and IPSec preference list with the configuration values that match the values configured on the remote gateway. If required, see procedure "Configuring IPSec Preference and Preference List" in Nortel Gateway Controller Administration and Security ( (NN10213-611)).
14
Change the connection policy selected in step 5 by selecting the new IPSec preference list. If required, see the appropriate steps in procedure "Modifying an existing IPSec connection policy" in Nortel Gateway Controller Administration and Security ( (NN10213-611)).
15
If the alarm does not clear within 5 minutes, contact your next level of support. Otherwise, the procedure is complete.
--End--
134
Changing the MAC address of a GWC card

This procedure describes how to automatically re-provision a known good GWC card with a new media access control (MAC) address. All other card information including IP addresses, port addresses, gateway addresses, and load paths remain unchanged. Use this procedure when you need to change the MAC address of a GWC card due to possible address conflicts or to ensure information in the CS 2000 SAM21 Manager database is correct.
Procedure steps
Step 1 Action At the CS 2000 Management Tools window, click the Gateway Controller folder from the Device Types directory tree in the far left frame. From the Contents of: Gateway Controller frame, select the GWC node for which you want to change the MAC address, or type the name of the GWC node in the text field to the left of the Go To button. If the GWC card you need to automatically re-provision is currently providing service, switch call processing between the two GWC units. Follow procedure "Invoking a manual protection switch (warm SWACT)" in Nortel Gateway Controller Administration and Security ( (NN10213-611)). 4 Busy the card that you want to re-provision (typically, the standby card in the node). Click the Maintenance tab, then click the Busy (Disable) button. When prompted, click OK.
Procedure steps
135
Click the Card View button for the card you just busied. The CS 2000 SAM21 Manager window opens.
Click the Lock button to lock the card.
Observe the History display to confirm that the card is locked. Look for the text "Application locked successfully". Also, notice the lock icon on the card graphic at the left of the screen and the Card Status "Locked".
136
At the SAM21 shelf
8 9 10
Remove the Ethernet and serial cables (if present) from the GWC faceplate. Open the ejector levers. Wait for the green LED on the faceplate to extinguish and the blue LED to appear at the bottom of the faceplate.
CAUTION
Possible service outage
A service outage can occur if care is not taken while removing the GWC circuit packs. The spiral gasket, located on the faceplate of the circuit pack, can become caught on an adjacent card and ripped off of the faceplate. If the spiral gasket ends up making contact with the backplane inside the chassis, an electrical short circuit can result in a service outage.
CAUTION
Static electricity damage
Wear an electrostatic discharge (ESD) grounding wrist strap connected to the SAM21 Shelf Cabinet when handling a GWC card. The strap protects the card against damage caused by static electricity.
11 12
Hold the GWC card by the latches and remove the card from the shelf. Examine the circuit pack before (re)inserting it into the SAM21 chassis to ensure that the spiral gasket is seated and not loose.
Procedure steps
137
13
Hold the same GWC card by the latches and reinsert the card into the shelf. Do not push on the faceplate to seat the card.
14 15 16
Secure the card by tightening the captive screws at the top and bottom of the panel. Replace the cables on the GWC card faceplate. In the shelf view or card view window, wait until the GWC card you are re-provisioning appears as follows:
138
The card should appear with the correct text label (GWC-<x>-UNIT-<y>) and in a locked state (note the lock icon at the bottom). After the card is inserted and connected, it passes through the following states, indicated in the shelf view or the card view:
The card first appears with the text label "No Service" and locked. A short time afterwards, the CS 2000 GWC Manager determines if the newly inserted card can support the current provisioning. If a GWC card was inserted, the display then changes from "No Service" to "GWC-<x>-UNIT-<y>" with the "?" icon just above the lock icon. When the card is configured for GWC service, the "?" icon is removed from the display. At this point, the card is ready to be unlocked (reprovisioned).

17
Re-provisioning with the new MAC address does not take effect until the card is unlocked and rebooted. Click the States tab to display the status of the GWC card. Click the Unlock button to unlock the GWC card. This causes the card to reboot and to be automatically reprovisioned. Observe the History display until the screen message "Bootloaded successfully" appears. If the card status does not display "Application unlocked successfully", click the Lock button in the card view and wait for the "Application locked successfully" message. Then, click the Unlock button again.
18 19
Procedure steps
139
If you are still unable to successfully unlock a GWC card, contact your next level of support.
--End--
140
Restarting or rebooting a GWC card

Restart a GWC card to stop all software processes on the GWC card, perform a hardware reset, and reload the GWC card software from the Core and Billing Manager (CBM). Reboot a GWC card and force a GWC to download and execute a software load from the Communication server 2000 Core Manager or CBM.
Prerequisites
To reduce the risk of service interruption, first busy the GWC applications on specific GWC nodes using the Communication server 2000 GWC Manager. For more information, see Nortel Gateway Controller Administration and Security ( (NN10213-611)).
Procedure steps
Step 1 Action At the Communication server 2000 SAM21 Manager client from the Shelf View, right-click the GWC card you want to reboot and select Card View.
Procedure steps
141
2 3
At the Card View, select the States tab. Click the Lock button to lock the card.
4 5 6
Wait until the Card Status is Locked and the History window indicates that: Application locked successfully. Click the Unlock button. Monitor the reboot process. Wait until the Card Status changes to Unlocked and the History window indicates Bootloaded successfully.
--End--
142
Restarting GWC card services

Restart GWC card services to stop and restart call processing and network services on a standby GWC card. Use this procedure on a GWC card to determine if a known fault is temporary or persistent and if the fault is limited to this particular GWC card.
Prerequisites
To reduce the risk of service interruption, ensure that the card you are about to restart is in standby status. If it is not, perform a call processing switch activity (SWACT) to change the card state from active to standby. For more information about invoking a manual protection switch (warm SWACT), see Nortel Gateway Controller Administration and Security (NN10213-611).
Procedure steps
Step 1 Action At the main Communication Server 2000 Management Tools window, click the Gateway Controller folder from the Device Types directory tree in the far left frame. From the Contents of: Gateway Controller frame, select the appropriate GWC node you wish to restart. Click the Maintenance tab. Locate the Activity State field for the GWC unit and determine which unit (card) is active (either unit 0 or unit 1) and which is in standby mode. Click the Busy (Disable) button for the GWC unit in a standby mode.
2 3 4
Procedure steps Figure 5 GWC node
143
6 7
At the confirmation box, click OK . Wait for the Communication server 2000 GWC Manager screen to update the Operational state of the card to disabled(2), which indicates that the card is busied. Click the RTS (Enable) button to return the card to service. After 30 seconds, the Administrative state field changes to unlocked and the Operational state field changes to enabled.
The state change is automatic. However, if necessary, you can refresh the screen. At the top of the Communication server 2000 GWC Manager screen, click the Windows menu and select Refresh GWC Status.
--End--
144
Replacing and re-provisioning a GWC card

Use this procedure when Nortel support personnel instructs you to replace a faulty GWC card and automatically provision the replacement card with the provisioning data of the previous card. During this process, the system assigns a new media access control (MAC) address to the replacement card but all other card information, including IP addresses, port addresses, gateway addresses, and load paths, remain unchanged. This procedure does not provide instructions to make services provisioning changes to a GWC card, such as changing the service profile type of a GWC node. If changes to the GWC node provisioning are necessary, use the CS 2000 GWC Manager to busy the cards and re-provision node information. For more information, see Nortel Gateway Controller Configuration ( (NN10205-511)).
CAUTION
MCPN750 card does not support the following GWC service profiles: AUDCNTL_RMGC, AUDCNTL_RMGCINTL, BICC, VRDN, VRDNINTL, H.323_INTL_V2, H.323_NA_V2, LARGE_LINEINTL_V2, LARGE_LINENA_V2, LINE_TRUN K_AUD_INTL, LINE_TRUNK_AUD_NA, SIP_T, SIP_TINTL, SMALL_LINEINTL_V2, SMALL_LINENA_V2 If the GWC node is configured with one of these profiles, do not replace the faulty card with an MCPN750 card. Otherwise, the card will be locked and will not provide service.
Prerequisites
If the card is not already busied (disabled), busy the card. If required, see procedure "Busying a GWC node" in Nortel Gateway Controller Configuration ( (NN10205-511)).
Procedure steps
Step 1 Action Right-click the card the card that you need to replace and select Card View.
Procedure steps
145
2 3 4 5 6
In the Card View, click the States tab. Click the Lock button to lock the card. Remove the Ethernet and serial cables (if present) from the GWC faceplate. Open the bottom ejector lever. Wait for the blue LED at the bottom of the faceplate to turn on, and wait for the red LED above the card to extinguish. (The red LED indicates that the card is out of service.)
At the SAM21 shelf
CAUTION
Possible service outage
A service outage can occur if are not careful while removing the GWC circuit packs. The spiral gasket, located on the faceplate of the circuit pack, can become caught on an adjacent card and ripped off of the faceplate. If the spiral gasket ends up making contact with the backplane inside the chassis, an electrical short circuit can result in a service outage.
146
CAUTION
Static electricity damage
Wear an electrostatic discharge (ESD) grounding wrist strap connected to the SAM21 Shelf Cabinet when handling a GWC card. The strap protects the card against damage caused by static electricity.
7 8
Press both ejector levers until card is ejected from the shelf. Examine the new circuit pack before inserting it into the SAM21 chassis to ensure that the spiral gasket is seated and not loose.
Hold the replacement GWC card by the latches and insert the card into the shelf. Do not push on the faceplate to seat the card.
Procedure steps
147
10 11 12
Secure the card by tightening the captive screws at the top and bottom of the panel. Replace the cables on the GWC card faceplate. Wait for a card icon with a hashed outline to appear in the shelf view. On insertion of the new card, the system automatically provisions the new card with the old cards provisioning information. The system assigns a new MAC address to the new card. The CS 2000 SAM21 Manager displays the card type (MCPN905, MCPN750, or CPCI6115) and the corresponding memory size in the Equip tab of the card view.
13
If the provisioning data for the replacement card is correct, the procedure is complete. If the provisioning data for the replacement card is not correct, go to the next step.
14
Open the Card View for the new card and click the Provisioning tab.
148
15 16
Click the Modify button. Enter the new or changed provisioning data, then click the Save button. You do not need to type the load name. Click the Get Load Files button and select the required load from the drop-down list.
17 18 19
Click the States tab. Click the Unlock button to unlock the card. Observe the History window to ensure that the card boot loaded and unlocked successfully.
--End--
149

This section provides information about Gateway Controller (GWC) service-related alarms, including appropriate actions to diagnose and resolve the alarm condition. This section also includes reference information related to V5.2 audit logs and to Kerberos, IPSec, and IKE security logs. For all other GWC log report descriptions, see Nortel Carrier Voice over IP Fault Management Logs Reference Volume 2 ( (NN10275-909v2)).
GWC service alarms (page 150) GWC V5.2 audit logs (page 166) IPSec and IKE security logs (page 170) Kerberos logs (page 177) GWC security logs (page 182)
150
GWC service alarms

This section contains details about GWC alarm types, including appropriate actions to diagnose and resolve the alarm condition. In the following table, an alarm ID code for each alarm appears in the first column under the alarm description. You can also find these logs with the alarm ID code in the following locations:
Integrated Element Management System (IEMS). For instructions on syslog forwarding in a network containing the IEMS, refer to the CS 2000 Management Tools information in Nortel ATM/IP Solution-level Fault Management ( (NN10408-900)).
The syslog customerlog files in the /var/log directory on the CS 2000 Management Tools server. For example, see the file customerlog.1. For information about how to open these syslog log files and search for alarm codes, see procedure Viewing GWC logs in syslog files (page 56).
These logs may also be available in syslog format in custlog files in the /var/adm directory on the Core and Billing Manager (CBM). The CS 2000 Management Tools server must be configured to send the syslog logs to the . If required, see the following procedures in Nortel ATM/IP Solution-level Fault Management ( (NN10408-900)):
"Routing log streams from an SPFS-based server to a remote host" "Configuring SESM log reporting"
Switch Control Center (SCC2) or Nortel STD log formats available on the customers Operations Support System (OSS) interface. Conversion to these log formats must be activated at the . For information about configuring log delivery to the customers OSS, see the Fault Management document for the .
Table 14 Troubleshooting GWC service alarms Alarm description Alarm ID code Severity Recovery alarm GWCEM301 Critical Specific problem Probable cause A GWC recovery process has terminated early, and the GWC remains out of service. This alarm is generated by the CS 2000 GWC Manager rather than the GWC.
Action Check IP communications from OAM system to the GWC. Check the CS 2000 Management Tools server logs for additional information, and BSY/RTS the affected GWC unit.
Procedure steps
151
Alarm description Alarm ID code Severity
Specific problem Probable cause Specific problem: Functionality not supported. This problem occurs when the GWC load is incorrect.
Action If the GWC load is incorrect, change the load, then lock and unlock the GWC card.
Active unit disabled GWC300 Critical
Specific problem: Indicates that a unit is out of service: Service is not available. Probable cause: the lack of availability of the underlying resource.
This alarm reports that the unit is not in service (Operational state of "disabled"). If the Administrative state is "unlocked" and the Usage state is "busy" this indicates the unit is trying to recover itself to an in service state. When both units are out of service, the active unit must recover before the standby unit will. Check the following:
Whether the unit is manually locked out of service (Administrative state of "locked"). Alarms that may indicate a problem on the unit preventing it from returning to service. Other state indicators which may indicate problems, such as: Isolation state of "isolated" Availability state of "offLine"
Logs which may also indicate a failure of a step in the process of recovering the unit.
152
Specific problem Probable cause Specific problem: Indicates that a unit has invalid GWC Profile Data: Service is not available. Probable cause: A configuration or customization error.
Action Check the profile data for the unit and do one of the following: - Change to another profile. - Reconfigure the GWC unit software or CS 2000 GWC Manager software to remove the incompatibility. Then RTS the unit.
Standby unit disabled GWC301 Major
Specific problem: Unit is out of service - Service is not available. Probable cause: The lack of availability of the underlying resource.
This alarm reports that the unit is not in service (Operational state of "disabled" and Administrative state is "unlocked"). The unit is system busy (SysB). If the Administrative state is "unlocked" and the Usage state is "busy" this indicates the unit is trying to recover itself to an in service state. When both units are out of service, the active unit must recover before the standby unit will. Check the following:
alarms that may indicate a problem on the unit preventing it from returning to service other state indicators which may indicate problems, such as: isolation state of "isolated" for the standby unit, availability state of "degraded" availability state of "offLine"
logs which may also indicate a failure of a step in the process of recovering the unit
Procedure steps
153
Alarm description Alarm ID code Severity Standby unit disable GWC301 Minor
Specific problem Probable cause Specific problem: Unit is out of service - Service is not available. Probable cause: The lack of availability of the underlying resource.
Action This alarm reports that the unit is not in service (Operational state of "disabled" and Administrative state is "locked"). The unit is manually busy (ManB). If the Administrative state is "unlocked" and the Usage state is "busy" this indicates the unit is trying to recover itself to an in service state. When both units are out of service, the active unit must recover before the standby unit will. Check the following:
alarms that may indicate a problem on the unit preventing it from returning to service other state indicators which may indicate problems, such as: isolation state of "isolated" for the standby unit, availability state of "degraded" availability state of "offLine"
Specific problem: Unit out of service - invalid GWC Profile Data. Probable cause: Configuration or customization error.
logs which may also indicate a failure of a step in the process of recovering the unit
Check the profile data for the unit and do one of the following: - Change to another profile. - Reconfigure the GWC unit software or CS 2000 GWC Manager software to remove the incompatibility. Then, RTS the unit.
154
Alarm description Alarm ID code Severity Core communication lost GWC302 Major, Minor Mate unit communic ation lost GWC303 Major, Minor Communication with a gateway is down GWC304 Major
Specific problem Probable cause Specific problem: No response received to Core heartbeat messages from active GWC unit. Probable cause: LAN error. Specific problem: No response received to mate unit heartbeat messages. Probable cause: LAN error. Specific problem: Communication with gateway <Gatewayname> is down. Probable cause: The underlying resource is not available. Gateway is defined as large gateway or audio gateway (e.g. PVG, H.323 gateway with 64 or more endpoints, MG 9000, UAS).
Action Clears automatically after a Core or network outage clears. Otherwise, verify that the node number and Core Side IP address is correct for the GWC to communicate with the Core.
Cleared by restoring communication from the CS 2000 GWC Manager to the GWC unit. Do this by unlocking the GWC at the CS 2000 SAM21 Manager. Also, verify that the Ethernet cable is connected, and that the GWC is setup to use the correct node number. Cleared by restoring communication to the managed gateway. Do this by verifying the availability of the gateway, and comparing the configuration data at the gateway and the CS 2000 GWC Manager (IP address, protocol/version, etc.).
This is a test alarm generated from pmdebug interface GWC305 Critical, Major, or Minor
Specific problem: Alarms test from debug interface. Probable cause: Unspecified reason.
Cleared from pmdebug command (not a customer interface) or with a GWC reload.
Procedure steps
155
Alarm description Alarm ID code Severity DQoS/COPS connection failure GWC306 Major
Specific problem Probable cause Specific problem: A DQoS connection <ConnN ame@node_ip_address> has failed - attempting recovery. Probable cause: Communications subsystem failure. See Fault management in a DQoS network (page 14) for details about what happens when a DQoS connection fails.
Action The DQoS connection loss alarm is cleared by DCCNXMGR (using DCALARM) when the connection is reestablished or the connection is deleted by provisioning.
DQoS/COPS connection failure GWC306 Critical
Specific problem: Connection failure alarm threshold - see customer log details Probable cause: Communications subsystem failure. See Fault management in a DQoS network (page 14) for details about what happens when a DQoS connection fails.
This alarm is generated when the alarm threshold is exceeded (more than 21 failure alarms). In addition to this critical alarm, each subsequent failing node also generates a customer log GWC503. When the number of failures decreases to below 12, the critical alarm is cleared. At this point, if log GWC503 was generated, the system generates log GWC502. For the description of logs GWC502 and GWC503, see Nortel ATM/IP Solution-level Fault Management ( (NN10408-900)).
Element Manager communication failure GWC307 Major
Specific problem: CS 2000 GWC Manager indicates provisioned data mismatched in this unit. Probable cause: Communications subsystem failure
Cleared with a Busy/RTS of GWC unit.
156
Specific problem Probable cause Specific problem: CS 2000 GWC Manager not responding, provisioned data loaded from local Flash. Probable cause: Communications subsystem failure
Action Restore communication with the CS 2000 GWC Manager. Determine if the CS 2000 GWC Manager is down or disconnected, or both. Determine if the GWC has been setup to use the wrong IP address for the CS 2000 GWC Manager at the CS 2000 SAM21 Manager.
Flash memory error GWC308 Minor
Specific problem: Erase of flash sector failed. Probable cause: Equipment malfunction. Flash life span exceeded; the number of writes to flash has exceeded the recommended or intended limit. Specific problem: The number of IPSec security associations (that is, secure connections) reached the maximum supported number. Probable cause: Resource at or nearing capacity
Cleared with hardware replacement.
SA_PERCENTAGE_ USAGE GWC309 Minor This alarm does not apply to the Wireline Universal Packet Access (UA-AAL1) solution. Provisioned GWC Profile not yet activated GWC311 Warning
This is an information alarm. Report this alarm with details to your next level of support. Note that the alarm clears automatically as SA usage decreases.
Specific problem: GWC profile loaded into Flash will activate on the next reload. A New GWC Profile has been loaded to GWC FLASH by the CS 2000 GWC Manager, but the GWC is still using the old Profile. Probable cause: Configuration or customization error.
Cleared with a GWC reload.
Procedure steps
157
Alarm description Alarm ID code Severity QoS collection application (QCA) connection failure GWC312 Major (partial outage)
Specific problem Probable cause Specific problem: QCA connection <ConnName> has failed attempting recovery. Probable cause: Communications subsystem failure.
Action No reports are lost since a back up server is collecting them. Check the following: 1. Ensure that the QCA contains the correct properties (port, IP address). Check that the QCA is properly provisioned using the CS 2000 Management Tools. 2. Use the ping command to see if you can reach the QCA server. If you cannot reach the server, there may be a problem in the network. 3. Verify that there is no memory exhaustion on the QCA server. 4. Restart the QCA application on the server to bring up the links. 5. Try connecting to a QCA on another CS 2000 Management Tools server.
QCA connection failure GWC312 Critical (total outage)
Specific problem: QCA connection <ConnName> has failed attempting recovery. Probable cause: Communications subsystem failure.
Check the following: 1. Ensure that the QCA contains the correct properties (port, IP address...). Check that the QCA is properly provisioned using the CS 2000 Management Tools. 2. Use the ping command to see if you can reach the QCA server. If you cannot reach the server, there may be a problem in the network. 3. Verify that there is no memory exhaustion on the QCA server. 4. Restart the QCA application on the server to bring up the links. 5. Try connecting to a QCA on another CS 2000 Management Tools server.
158
Alarm description Alarm ID code Severity RMGC overloaded GWC313 Major
Specific problem Probable cause Specific problem: RMGC cant process all incoming requests. Probable cause: Resource at or nearing capacity.
Action The RMGC is temporarily overloaded The alarm will clear itself once the RMGC is able to process requests again. Gateways keep sending requests until they get a response. So, once the overload clears, gateways will be able to register without any further intervention. If this alarm is seen regularly or does not clear, then this is an indication that there is insufficient RMGC processing capacity in the office. Consider commissioning another RMGC.
Location ID reporting connection failure GWC314 Major
Specific problem: Location ID reporting connection <IP address> has failed - attempting to recover. Probable cause: Communications subsystem failure - destination not available. Specific problem: Small GWs in disabled state Probable cause: Signal loss - alarm based on number of GWs out of service compared with number of GWs provisioned
Clear the alarm condition using one of the following approaches:
Reestablish the connection to the location recipient. Disable the location ID reporting application. Busy/RTS the GWC unit.
External host interface signal loss GWC315 Critical (>50% loss) Major (>20% loss) Minor (any signal loss)
Logs (GWC502) indicate a small line GW going out of service and coming back into service. GWs are considered to be out of service when heartbeat messages between the GW and GWC are lost. Use the new small gateway OMs to check how many small line GWs are out of service. Use the logs to identify which GWs need attention.
Procedure steps
159
Alarm description Alarm ID code Severity External host interface signal loss GWC315 Major (both links down) Minor (one link down) External host interface signal loss GWC315 Critical (>50% loss) Major (>20% loss) Minor (any signal loss) External host inter face configuration error GWC315 Critical (50% or more) Major (7% or more) Minor (1% to 6%) Warning (<1%)
Specific problem Probable cause Specific problem: USP SS7 paths disabled Probable cause: Signal loss
Action This alarm is based on the number of USP connections that are out of service, compared with the number of paths (maximum two) checked in a 5-minute polling period. The link paths are considered to be out of service when heartbeat messages between the GWC and USP are lost. Check the USP connections.
Specific problem: Peer connections failed during interval Probable cause: Signal loss
This alarm is based on the number of call setups failed, compared with the number of calls attempted in a 5-minute polling period. Use the new peer messaging OMs to check the number of calls failed over a longer period. Check the Ethernet link between the GWC and the remote host or peer GWCs.
Specific problem: DNS failed GW discovery Probable cause: Configuration or customization error
This alarm is based on the number of gateways that failed DNS lookup in a 5-minute polling period. Use the new DNS OMs to check the number of gateways not discovered, and the number of gateways discovered using RSIP. RSIP discovery does not survive a cold SwAct on the GWC, which may cause long outages. Check the DNS server entries for every gateway.
160
Alarm description Alarm ID code Severity External host interface protocol error GWC315 Major (both links not active) Minor (one link not active) External communicat ion interface GWC315 Critical (>50% proxies disabled) Major (>20% proxies disabled) Minor (any media proxies disabled)
Specific problem Probable cause Specific problem: USP SS7 path not active Probable cause: Communications protocol error
Action This alarm is based on the number of USP connections that have heartbeat messages but are not active in a 5-minute polling period. A major alarm affects some call processing; a minor alarm causes a redundancy error. Check the USP configuration to the GWC.
Specific problem: Media proxies in disabled state Probable cause: Signal loss - alarm based on number of media proxies out of service compared with number of media proxies provisioned
Media proxies are considered to be out of service when they fail to respond to messages sent from the GWC, and returned to service when the GWC receives the regular RSIP message. The alarm is raised as a media proxy goes out of service, and cleared when it comes back into service. Use the media proxy OMs to check how many media proxies are out of service in any 5-minute polling period. Use the RTP Portal EM to identify which media proxies need attention. This alarm and the associated OMs give the GWC view of calls, as opposed to the media proxy view.
PreSwact audit failure GWC317
Specific problem: PreSwact audit failed for two consecutive cycles Probable cause:
Check the alarm text to see which component has failed. The alarm clears when you clear the error condition. If required, you can use Swact force to force a manual warm Swact.
Major
monitored base resource reaches the threshold level monitored application resource reaches the threshold level
Procedure steps
161
Specific problem Probable cause
Action
data synchronization mismatch (SESM, GWC flash, GWC RAM) standby unit is jammed and prevented from taking up the activity
The preSwact audit runs at a frequency of 40 seconds and with priority 6. A second occurrence of the same error condition does not raise an alarm. No alarm is raised if the preSwact audit fails while patching is in progress in the inactive unit. Autonomous SWACT initiated GWC318 Critical Specific problem: <Element Name> No communication with associated media gateway(s) Probable cause: Underlying resource unavailable. This alarm indicates that the system attempts to restore communication with the associated media gateways. GWC communi cation loss with associated media gateway(s) GWC319 Critical Specific problem: <Element Name> No communication with associated media gateway(s) Probable cause: Underlying resource unavailable. Check the related network component managers to identify failures within the network. The alarm clears automatically when a manual SWACT back to the original active unit is performed. You can also manually clear this alarm once the problem on the original active unit is solved. Follow procedure Clearing the GWC318 critical alarm manually . Check the related network component managers to identify failures within the network. This alarm clears when the GWC detects communication with at least one associated gateway. It also clears when the GWC automatically or manually switches activity, but it is raised again on the newly active unit if communication is still lost with all associated gateways.
IKE/IPSec GWC320
GWC320 alarms indicate an IKE/IPSec-related problems. See the Specific Problem: field to check the reason for generating this alarm. Each specific alarm may have an associated security log, which provides additional details about the problem. .
162
Alarm description Alarm ID code Severity GWC320 Critical - certificate expires within 5 days Major - certificate expires within 15 days Minor - certificate expires within 30 days
Specific problem Probable cause Specific problem: One or more certificates in set number <x> (device cert serial # <xxx>) are expiring. An outage may occur if an IKE connection policy is using these certificates. At the GWCEM, please verify the expiry dates for certificates in set number <x> -- <GWC_IP_address> where <x> is the set number of the [IKE certificates] panel <xxx> is the serial number of the expiring certificate Probable cause: Key expired
Action This alarm indicates that an IKE certificate is expiring soon and the GWC Manager has not obtained a new certificate from the Certificate Manager. Service disruption may occur when the certificate expires. For information about how to clear this alarm, see procedure Recovery of the GWC320 certificate expiry alarm in Nortel CVoIP IPSec Service Configuration ( (NN10453-100)).
GWC320 Critical
Specific problem: One or more certificates in set number <x> (device cert serial # <xxx>) are expired. An outage will occur if an IKE connection policy is using these certificates. At the GWCEM, please verify the expiry dates for certificates in set number <x> -- <GWC_IP_address> where <x> is the set number of the [IKE certificates] panel <xxx> is the serial number of
This alarm indicates that an IKE certificate is expired and the GWC Manager has not obtained a new certificate from the Certificate Manager. Service disruption will occur when the certificate expires. For information about how to clear this alarm, see procedure Recovery of the GWC320 certificate expiry alarm in Nortel CVoIP IPSec Service Configuration ( (NN10453-100)).
Procedure steps
163
Specific problem Probable cause the expiring certificate Probable cause: Key expired
Action
GWC320 Critical
Specific problem: Phase 1 SA failure -<remote_gateway_IP _address> Probable cause: Underlying resource unavailable
This alarm indicates that a Phase 1 negotiation failed. This alarm is the result of an outage. To clear this alarm, complete procedure Clearing GWC320 Phase 1 SA failure alarm. This alarm indicates that a Phase 2 negotiation failed. This alarm is the result of an outage. To clear this alarm, complete procedure Clearing GWC320 Phase 2 SA failure alarm.
GWC320 Critical
Specific problem: Phase 2 SA failure -<remote_gateway_IP _address>
Probable cause: Underlying resource unavailable GWC320 Critical Specific problem: Private key doesnt match GWC certificate in set number <x>(device cert serial # <xxx> -- <GWC_IP_address> where <x> is the set number of the [IKE certificates] panel <xxx> is the serial number of the certificate Probable cause: Unexpected Information
This alarm may indicate an outage. Contact your next level of support. There are no security logs associated with this specific problem.
164
Alarm description Alarm ID code Severity GWC320 Critical
Specific problem Probable cause Specific problem: Invalid Subject Alternative Name in GWC certificate set number <x> (device cert serial # <xxx>) -<GWC_IP_address> where <x> - is the set number of the [IKE certificates] panel <xxx> is the serial number of the certificate Probable cause: unexpectedInformation
Action This alarm indicates that the Subject Alternative Name in the GWC certificate does not match the provisioned IP address for the GWC certificate. This may indicate a data or communication problem between the GWC and the GWC Manager. Contact your next level of support.
Ethernet/IP Errors GWC321 Major
Specific problem: Auto-negotiation disabled on Ethernet link. Probable cause: Configuration or customization error Specific problem:
Enable the auto-negotiation capability on the Ethernet Routing Switch 8600. If required, see the Ethernet Routing Switch 8600 configuration documentation.
A subsystem overloaded GWC322 Major - SNMP message rate crossed the Panic Level Warning - SNMP message rate crossed the Warning Level
Reduce the SNMP traffic level.
for major alarms:
SNMP message rate exceeded recommended maximum
for warning alarms:
SNMP message rate is close to recommended maximums Probable cause: Threshold crossed
Procedure steps
165
Alarm description Alarm ID code Severity Communication with a gateway is down GWC501 Major
Specific problem Probable cause Specific problem: Gateway failed to respond to heartbeat/audit Probable cause: The underlying resource is not available. Gateway is defined as small gateway (e.g. Askey/Mediatrix line gateway, Arris/Motorola gateway, H.323 gateway with less than 64 endpoints).
Action Cleared by restoring communication to the managed gateway. Do this by verifying the availability of the gateway, and comparing the configuration data at the gateway and the CS 2000 GWC Manager (IP address, protocol/version, etc.).
Data inconsistency between GWCEM and Session Server Minor
Specific problem: Data inconsistency between GWCEM and Session Server Probable cause: Depends on the condition that caused the alarm. Examples: "Data Audit run but data not fully synchronized", "Setting the Call Agent ID has resulted in data changes, which need to be propagated to the Session Server. Run the CS2KSS EM Data Integrity Audit", "Failed to rollback add zone command from Database", "Failed to rollback delete zone command.
For all cases other than when the Data Audit fails, run the CS2KSS Data Integrity Audit. If the data audit fails, investigate the reason (for example, connection failure, problem with the data on the CS2KSS EM, and so on) and manually correct the problem.
166
GWC V5.2 audit logs

This section describes troubleshooting activities on the GWC or a related component in response to a particular V5.2 fault log. There are a number of GWC-related log types that are specific to V5.2 lines maintenance. V5.2 audit logs are generated to capture regular maintenance test results made to associated V5.2 media gateways and access nodes. The BCC audit verifies a possible mismatch between GWC and AN while the V5CC audits verify a possible mismatch CM and GWC. See the relationship as follows:
V5 BCC Audit V5CC Audit
V5 Interface Audit V5 Link Audit V5 C-channels Audit V5 Data Link Audit

Use the following table to review the details of the log type displayed by your log utility and formulate the appropriate actions to diagnose and repair the problem. For more information about V5.2 logs, see the appropriate V5.2 log description in Nortel Carrier Voice over IP Fault Management Logs Reference Volume 2 ( (NN10275-909v2)).
Table 15 V5.2 logs Log type; Problem logged V5200 (BCC Audit fails) V5201 (BCC Audit message)
Reason for failure Generated when an AN (access node) does not respond to a BCC Audit message. Generated when an audit message is sent from the CM to an access node.
Details The BCC audit allows checking of a possible mismatch between GWC and access node (AN). It is executed when the AN sends a BCC Reject message to the GWC, upon receiving BCC Allocation. There are several reject causes, which are
Procedure steps
167
Log type; Problem logged V5202 (BCC Audit incomplete)
Reason for failure Generated during a BCC (bearer channel control) audit, when the returned "Audit Complete" message includes the information element "Connection Incomplete".
Details given in the BCC Reject message. Some of those reject causes will make the GWC send a BCC Audit to the AN.
Connection already present at the PSTN user port to a different V5 time slot (0x83) Connection already present at the V5 time slot(s) to a different port or ISDN user port time slot (0x84) Connection already present at the ISDN user port time slot(s) to a different V5 time slot(s) (0x85) Deallocation cannot be completed due to V5 time slot(s) data incompatibility (0x88) Deallocation cannot be completed due to port data incompatibility (0x89) Deallocation cannot be completed - user port time slot(s) data incompatibility (0x8A)
V5400 (V5CC Audit)
Generated when there is no reply from the V5 interface for the V5 audit queries during V5CC audit.
V5CC interface audit is the only audit executed if an interface in the deactivated status. The V5CC (channel control) Audit performs consistency checks for various interface, link and line statuses. When the GWC is not in service, when a GWC startup/activation process takes place or when in a GWC is in a maintenance in-progress state, a V5CC audit will not be executed. V5 interfaces will be audited in the following order:
V5 interface audit V5 link audit V5 c-channel audit V5 data link audit
168
Log type; Problem logged
Reason for failure
Details
V5 babbling lines audit V5 line state audit
An interface will receive audit queries every 10 minutes. V5401 (V5 Interface Audit) Generated when a mismatch is detected by the CM for a queried V5 interface on a GWC. During a V5CC audit, the CM sends a V5 interface query to an interface on a GWC, the query message requests the status of the interface on the GWC. The GWC will send a response message upon receiving a V5 interface query message with the status of the corresponding interface (either ACT or DEACT). If a mismatch is detected, the CM will request the GWC to change the status of the interface to that held on the CM. V5402 (V5 Link Audit) Generated when a mismatch is detected by the CM for a queried V5 link on a GWC. During a V5CC audit, a V5 link audit is performed. When the GWC sends a reply message, which contains the status of the links, the CM will check the status of the link carrier and compare it with the status of a carrier flag. In case of mismatch of carrier flag, the CM will send a message to GWC in order to open or close scanning on the given link, respectively. V5403 (V5 C-channels Audit) Generated when a mismatch occurs between the status of the C-channel as recorded in the GWC and in the CM. The CM sends a V5 C-channel audit to a GWC to request the C-channel information on the GWC. No action is taken on the GWC side. After receiving the response message from GWC, the CM looks for the following:
C-channel status mismatch (INSV/OOS) C-channel activity mismatch (ACT/STBY) C-channel static data mismatch
Procedure steps
169
Log type; Problem logged
Reason for failure
Details In all cases of mismatch, the C-channel status on the CM will be updated according to the status held on the GWC. No additional maintenance request is needed upon mismatch detection.
V5404 (Data Link Audit)
Generated when a mismatch occurs between the status of the data links as recorded in the GWC and in the CM.
The CM sends a V5 data link audit message to a GWC to request a data link status on the GWC. No action is taken on the GWC side. The GWC will send a response message which contains the current data link status. The data link status consists of CTRL, PSTN, BCC, LNK_CTRL, PROT1 and PROT2 statuses. After receiving a response message from GWC, the CM will look for a mismatch, In case of mismatch, the appropriate alarm status on the CM will be updated according to the status of the data link. Additional maintenance requests to reset (MANRTS) or block (MANBSY) all V5 lines is sent upon mismatch detection. Log V5404 will be generated in case of a mismatch.
170
IPSec and IKE security logs

This section describes how to access and understand IPSec and IKE security log reports associated with GWC nodes. These log reports are stored in the securitylog files in directory /var/log on the CS 2000 Management Tools server. To access these security log reports, follow procedure Viewing GWC logs in syslog files (page 56). To display IPSec and IKE log reports, search for the text string ISAKMP within the securitylog files.
Format
The format for security log reports is as follows:
Selected eld descriptions

The following table explains selected fields in the log report.
Field mmm dd hh:mm:ss Value alphanumeric Description The date and time stamp for the log report. mmm means three first letters of the month, for example, Aug. <IP address> <log type> ISAKMP_INFO or ISAKMP_FAIL <log message> numeric class_security.ver01 text string The IP address of the unit device from which the log originated. Common name for security logs. Indicates an information-only or a problem-reporting log. A description of the conditions or reasons generating the log. For log descriptions and associated actions, see section Actions (page 171). Indicates whether the operation succeeded or failed. 0016 indicates an INFO log. 0017 indicates a FAIL log.
alphanumeric character string
<status> <MID>
success or failure MID=MID_Nortel_MGC_autho.<0016 or 0017>
Actions
171
Actions
Required action depends on the log description. The following table lists security log messages and explanations, and the appropriate actions.
Table 16 IPSec and IKE security logs Log message Explanation and Action
Logs associated with the "Phase 1 SA failure" GWC320 critical alarm:
ATTENTION
Service disruption
The following log reports and the associated alarms are the result of an outage. To clear these fault conditions, complete procedure Clearing the GWC320 Phase 1 SA failure alarm (page 128). ISAKMP_FAIL IKE phase 1 timeout exhausted (PHASE 1 <INITIATOR or RESPONDER>, init:<IP>, resp:<IP>) ISAKMP_FAIL IKE payload not formed correctly, possible preshared key mismatch (PHASE 1 INITIATOR, init:<IP>, resp:<IP>) ISAKMP_FAIL IKE payload validation failed, possible preshared key mismatch (PHASE 1 <INITIATOR or RESPONDER>, init:<IP>, resp:<IP>) ISAKMP_FAIL payload malformed, possible pre-shared key mismatch (resp:<IP>, init:<IP>) ISAKMP_FAIL payload malformed (resp:<IP>, init:<IP>) ISAKMP_FAIL No Preferences Match for IKE Phase 1 Negotiation (init:<IP>, resp:<IP>) ISAKMP_FAIL invalid signature (init IP:<IP>, resp IP:<IP>) ISAKMP_FAIL IKE phase 1 signature or certificate check (PHASE 1 <INITIATOR or RESPONDER>, init:<IP>, resp:<IP>) Phase 1 Error: Received a NO_PROPOSAL _CHOSEN message init:<IP>, resp:<IP>) Phase 1 Error : Received a PAYLOA D_MALFORMED message (init:<IP>, resp:<IP>) There is a problem during an IKE Phase 1 negotiation and multiple retransmission attempts fail. The system cannot authenticate the remote gateway.
The preshared key configured on the GWC does not match the preshared key configured on the remote gateway. The pre-shared key configured on the GWC does not match the pre-shared key configured on the remote gateway. The system cannot authenticate the remote gateway. IKE preferences configured on the GWC and on the remote gateway do not match. The GWC cannot authenticate the remote gateway. There is a problem during an IKE Phase 1 negotiation and the GWC cannot verify or authenticate the certificate presented by the remote gateway. IKE preferences configured on the GWC and on the remote gateway do not match. The system cannot authenticate the remote gateway. Check that both sides have the same preshared key or valid certificates.
172
Table 16 IPSec and IKE security logs (contd.) Log message Phase 1 Error : Discard Policy Provisioned (init : <IP>, resp : <IP>) Phase 1 Error : Bypass Policy Provisioned (init : <IP>, resp : <IP>) Phase 1 Error : Kerberos Policy Provisioned (init : <IP>, resp : <IP>) Phase 1 Error while processing SA Payload (init : <init_ip>, resp: <resp_ip>:) Phase 1 Error while processing HASH Payload (init : <init_ip>, resp: <resp_ip>:) Phase 1 Error while processing KE Payload (init : <init_ip>, resp: <resp_ip>:) Phase 1 Error while processing NONCE Payload (init : <init_ip>, resp: <resp_ip>:) Phase 1 Error while processing ID Payload (init : <init_ip>, resp: <resp_ip>:) Phase 1 Error while constructing SA Payload (init : <init_ip>, resp: <resp_ip>:) Phase 1 Error while constructing HASH Payload (init : <init_ip>, resp: <resp_ip>:) Phase 1 Error while constructing KE Payload (init : <init_ip>, resp: <resp_ip>:) Phase 1 Error while constructing NONCE Payload (init : <init_ip>, resp: <resp_ip>:) Explanation and Action A Discard Policy is provisioned for this gateway.
A Bypass Policy is provisioned for this gateway.
A Kerberos Policy is provisioned for this gateway. Unable to interpret or parse an IKE exchange message. Check the peer for error messages. Unable to interpret or parse an IKE exchange message. Check the peer for error messages. Unable to interpret or parse an IKE exchange message. Check the peer for error messages. Unable to interpret or parse an IKE exchange message. Check the peer for error messages. Unable to interpret or parse an IKE exchange message. Check the peer for error messages. Unable to create an IKE exchange message. The most likely cause is insufficient resources on the GWC. Contact your second level of support. Unable to create an IKE exchange message. The most likely cause is insufficient resources on the GWC. Contact your second level of support. Unable to create an IKE exchange message. The most likely cause is insufficient resources on the GWC. Contact your second level of support. Unable to create an IKE exchange message. The most likely cause is insufficient resources on the GWC. Contact your second level of support.
Actions
173
Table 16 IPSec and IKE security logs (contd.) Log message Phase 1 Error while constructing ID Payload (init : <init_ip>, resp: <resp_ip>:) Explanation and Action Unable to create an IKE exchange message. The most likely cause is insufficient resources on the GWC. Contact your second level of support.
Logs associated with the "Phase 2 SA failure" GWC320 critical alarm:
ATTENTION
Service disruption
The following log reports and the associated alarms are the result of an outage. To clear these fault conditions, complete procedure Clearing the GWC320 Phase 2 SA failure alarm (page 131). ISAKMP_FAIL No Preferences Match for IKE Phase 2 Negotiation (init IP:<IP>, resp IP:<IP>) ISAKMP_FAIL IKE phase 2 timeout exhausted (PHASE 2 <INITIATOR or RESPONDER>, init:<IP>, resp:<IP>) Phase 2 Error: Received a NO_PROPOSAL _CHOSEN message init:<IP>, resp:<IP>) Phase 2 Error while processing SA Payload (init : <init_ip>, resp: <resp_ip>:) Phase 2 Error while processing HASH Payload (init : <init_ip>, resp: <resp_ip>:) Phase 2 Error while constructing SA Payload (init : <init_ip>, resp: <resp_ip>:) Phase 2 Error while constructing HASH Payload (init : <init_ip>, resp: <resp_ip>:) IPSec preferences configured on the GWC and on the remote gateway do not match. There is a problem during an IPSec Phase 2 negotiation and multiple retransmission attempts fail. IPSec preferences configured on the GWC and on the remote gateway do not match. Unable to interpret or parse an IKE exchange message. Check the peer for error messages. Unable to interpret or parse an IKE exchange message. Check the peer for error messages. Unable to create an IKE exchange message. The most likely cause is insufficient resources on the GWC. Contact your second level of support. Unable to create an IKE exchange message. The most likely cause is insufficient resources on the GWC. Contact your second level of support.
Logs associated with the "Certificate expiring" GWC320 minor, major, or critical alarm:
174
Table 16 IPSec and IKE security logs (contd.) Log message Certificate is expiring on Year=<yyyy > Month=<mm> Day=<dd> Explanation and Action After an IKE Phase 1 negotiation, if any certificate in the chain (excluding the root CA) is expiring within 30 days, this failure log is generated. If the certificates are not replaced immediately, there may be a call processing outage. This log is associated a GWC320 alarm:
critical - certificate expires within 5 days major - certificate expires within 15 days minor - certificate expires within 30 days
If the certificate is expired, this alarm remains, and if any connection policy and IKE authentication is using this certificate, a "Phase 1 SA failure" alarm is also raised. To clear the "Certificate expiring" alarm, complete procedure Recovering of the GWC320 certificate expiry alarm in Nortel CVoIP IPSec Service Configuration ( (NN10453-100)). Logs associated with the "Certificate expired" GWC320 critical alarm: One or more certificates in set number <x> (device cert serial # <xxx>) are expired One or more certificates in a set is expired. An outage will occur if an IKE connection policy is using these certificates. To clear the "Certificate expired" alarm, complete procedure Recovery of the GWC320 certificate expiry alarm in Nortel CVoIP IPSec Service Configuration ( (NN10453-100)). Logs indicating problems with the Diffie-Hellman (DH) exchange - no associated alarms:
ATTENTION
Service disruption
The following log reports are the result of an outage. ISAKMP_FAIL IKE Phase 1 key negotiation failed - DH is NULL (ini:<ip>, resp:<ip>) The GWC cannot allocate sufficient resources. Contact your next level of support
Actions
175
Table 16 IPSec and IKE security logs (contd.) Log message ISAKMP_FAIL IKE Phase 1 key negotiation failed - unable to set peers DH pub key (ini:<ip>, resp:<ip>) ISAKMP_FAIL IKE Phase 1 key negotiation failed - unable to gen DH key pair (ini:<ip>, resp:<ip>) ISAKMP_FAIL IKE Phase 1 key negotiation failed - unable to compute shared secret (ini:<ip>, resp:<ip>) ISAKMP_FAIL DOI no supported (ini:<ip>, resp:<ip>) Explanation and Action The GWC cannot allocate sufficient resources. Contact your next level of support The GWC cannot allocate sufficient resources. Contact your next level of support The GWC cannot allocate sufficient resources. Contact your next level of support The GWC cannot negotiate a security association with the remote gateway. Contact your next level of support. Logs indicating problems with the Phase 1 exchange - no associated alarms:
ATTENTION
Service disruption
The following log reports are the result of an outage. To clear these fault conditions, complete procedure Clearing the GWC320 Phase 1 SA failure alarm (page 128). ISAKMP_FAIL invalid signature (ini:<ip>, resp:<ip>) ISAKMP_FAIL invalid payload (ini:<ip>, resp:<ip>) ISAKMP_FAIL invalid hash info (ini:<ip>, resp:<ip>) Information-only logs: Certificate provisioning - validity period checked: Certificate <effective or expire> Y/M/D H:M:S: <effective or expiry date and time> When a certificate is provisioned on the GWC, the GWC checks the certificate effective and expiry date and generates this log. When certificate provisioning fails, use this log to determine if the certificate is not effective or has expired. There is no associated alarm. The GWC cannot authenticate the remote gateway. The GWC cannot authenticate the remote gateway. The GWC cannot authenticate the remote gateway.
176
Table 16 IPSec and IKE security logs (contd.) Log message Phase 1 SA Successfully Established (init IP:<IP>, resp IP:<IP>) Explanation and Action This log confirms a successful IKE Phase 1 negotiation between the GWC and a remote gateway. If a GWC320 alarm is generated, it clears automatically. No action required. Phase 2 SA Successfully Established (init IP:<IP>, resp IP:<IP> This log confirms a successful IPSec Phase 2 negotiation between the GWC and a remote gateway. If a GWC320 alarm is generated, it clears automatically. No action required.
Associated OM registers
These log reports have no associated OM registers.
Additional information
None
Action
177
Kerberos logs
This section describes how to access and understand log reports associated with the Kerberos application running on the GWC card. These log reports are stored in the securitylog files in directory /var/log on the CS 2000 Management Tools server. To access the Kerberos log reports, follow procedure Viewing GWC logs in syslog files (page 56). To display Kerberos log reports, search for the text string KERBEROS (common name for all Kerberos logs). You can also access the Kerberos log reports through the Integrated Element Management System (IEMS). For more information, see Nortel IEMS Fault Management ( (NN10334-911)).
Format
The format for Kerberos log reports is as follows:
Selected eld descriptions

The following table explains selected fields in the log report.
Table 17 Log field descriptions Field mmm dd hh:mm:ss Value alphanumeric Description The date and time stamp for the log report. mmm means three first letters of the month, for example, Aug. <host name> KERBEROS <log description> numeric text string alphanumeric character string The IP address of the GWC. Common name for all Kerberos log reports. A description of the conditions or reasons generating the log. The log can be static or variable. For log descriptions, causes, and associated actions, see section Action (page 177). The IP address of the remote gateway.
<remote IP address>
numeric
Action
The following tables list the static and variable Kerberos logs. Use these tables to determine your action.
178
Table 18 GWC Kerberos static logs Kerberos application log description WAKE_UP timeout after %d ms, exhausted after %d retries AP_REP timeout after %d ms, exhausted after %d retries AP_REP timeout after %d ms, retry attempt is now %d failed to get FQDN Cannot exceed maximum of %d KM sessions Received AP_REQ while waiting for SA_RECOVERED unsolicited SA_RECOVERED Cause or condition gateway fails to respond to WAKE_UP request gateway fails to respond to AP_REP (a request for a security association) gateway fails to respond to AP_REP (a request for a security association) gateway is not provisioned at the GWC a large number of gateways try to recover or restore connectivity at once race condition, or gateway did not receive AP_REP request gateway sends an SA_RECOVERED message. Possible cause is the gateway is responding to an old AP_REP (the session was deleted on the GWC). gateway sends an SA_RECOVERED message whereas the server didnt ask for it. gateway sends an SA_RECOVERED message whereas the server was not waiting for this message type race condition, an AP_REQ was initiated by the GW at the same time that a WAKE_UP was sent from the GWC Action verify connectivity between the GWC and the gateway verify connectivity between the GWC and the gateway verify connectivity between the GWC and the gateway verify gateways authenticity and provision gateway information-only log
information-only log
Received SA_RECOVERED while responder for existing key neg Received SA_RECOVERED out of order
CMS nonce is zero in AP_REQ reply to WAKE_UP
Some log descriptions use variables such as %d or %s to indicate a numeric value is provided.
Action
179
Table 18 GWC Kerberos static logs (contd.) Kerberos application log description CMS nonce mismatch in AP_REQ reply to WAKE_UP Cause or condition race condition, an AP_REQ was sent as a response to a previously initiated WAKE_UP race condition, an AP_REQ was sent by the GW as a response to a WAKE_UP after the WAKE_UP had already timed out Action information-only log
Non-zero CMS nonce in initiator AP_REQ
For all the following static logs, contact your next level of support: MUTUAL REQUIRED not set in AP_REQ USE_SESSION_KEY (not supported) set in AP_REQ Sub-key in AP_REQ is not allowed IP mismatch: fqdn=%s, ip=%s Failed HMAC in SA_RECOVERED NULL session key parsing AP_REQ but no KRB ERROR Some log descriptions use variables such as %d or %s to indicate a numeric value is provided. Table 19 GWC Kerberos variable logs Kerberos application log description The following log reports can be displayed with different <reasons>. For the list of possible reasons and the associated actions, see Table 20 "Kerberos log reasons" (page 180). <reason> while making KRB_ERROR message <reason> while checking AP_REQ proposal <reason> while generating AP_REP sub-key <reason> while adding pending incoming SA for AP_REQ <reason> while adding pending outgoing SA for AP_REQ <reason> while computing SA_RECOV HMAC <reason> while committing SAs for AP_REQ <reason> while parsing AP_REQ <reason> while parsing KRB_AP_REQ <reason> while verifying AP_REQ <reason> while parsing SA RECOV
180
Table 19 GWC Kerberos variable logs (contd.) Kerberos application log description <reason> while verifying SA RECOV <reason> while updating CLOCKSKEW <reason> when parsing name \"%s\" <reason> while updating server principal Table 20 Kerberos log reasons Kerberos log reasons "No IPSEC policy match" "IPSEC ciphersuite is not supported" "No policy match for AP_REQ" "Invalid SA lifetime" Action verify provisioning datafill; if required, configure an appropriate connection policy verify encryption and authentication provisioning datafill verify provisioning datafill verify provisioning datafill; make sure that the same values are configured on the GWC and the gateway verify provisioning datafill (encryption and authentication algorithms); make sure that the same values are configured on the GWC and the gateway verify provisioning datafill verify provisioning datafill verify provisioning datafill verify provisioning datafill synchronize the time between the GWC and KDC synchronize the time between the GWC and the gateway no action required - gateway should automatically request a new ticket. If reoccurring, check the KDC status and configuration. information only information only information only
"Invalid ciphersuite"
"No IPEC policy" "Invalid IPSEC proposal" "Invalid key length" "Invalid renewal period" "Ticket not yet valid" "Clock skew too great" "Ticket expired"
"Generic KRBKMP error "Message out of order" "Generic error (see e-text)"
For all other <reasons>, contact your next level of support.
181
Associated OM registers
This log report has no associated OM registers.
This log report has no additional information.
182
GWC security logs

This section contains the following Gateway Controller (GWC) security logs:
Telnet logs
Telnet logs (page 182) FTP logs (page 183) Serial connection logs (page 184)
This section contains GWC security telnet logs.
Login successful
Sample log: Jun 12 12:27:50 [172.16.16.42.128.192] class_security.ver 01 CMD=connect STAT=start SRC=172.16.16.4 CONNECT=telnet_connect MID=MID_Nortel_MGC_auth.0001 Jun 12 12:27:55 india1sesm SuccessionLogin: UserId:ptm\ n Status:Login Successful\n Message:User Granted Access Jun 12 12:27:55 [172.16.16.42.128.192] class_security.ver 01 DOC=login STAT=success SRC.USR=ptm MID=MID_Nortel_MGC_auth.0011
Login failure
Sample log: Jul 27 01:49:40 nc0rtp259.us.nortel.com class_security.ve r01 CMD=connect STAT=start TTY=0 MID=MID_Nortel_MGC_auth.0001 Jul 27 01:49:43 nc0rtp223.us.nortel.com GWC: class_securit y.ver01 CMD=connect STAT=start SRC=47.142.134.170 CONNECT=telnet_connect MID=MID_Nortel_MGC_auth.0001 Jul 27 01:49:47 nc0rtp2170 SuccessionLogin: UserId:rt\n Status:Login Failure\n Reason:Invalid UserName/Password Jul 27 01:49:47 nc0rtp223.us.nortel.com GWC: class_securit y.ver01 DOC=login STAT=failure SRC.USR=rt MID=MID_Nortel_MGC_auth.0011
Logout
Sample log: Jul 27 01:39:39 nc0rtp223.us.nortel.com GWC: class _security.ver01 CMD=logout STAT=end SRC.USR=rtp2 SRC=47.142.134.170
FTP logs
183
MID=MID_Nortel_MGC_auth.0002 Jul 27 01:39:39 nc0rtp223.us.nortel.com GWC: class_securit y.ver01 CMD=disconnect STAT=end SRC=47.142.134.170 MID=MID_Nortel_MGC_auth.0008
Forceout through another telnet connection

Sample log: Jul 27 01:48:25 nc0rtp223.us.nortel.com GWC: class_securit y.ver01 CMD=connect STAT=start SRC=47.142.134.170 CONNECT=telnet_connect MID=MID_Nortel_MGC_auth.0001 Jul 27 01:48:31 nc0rtp2170 SuccessionLogin: UserId:rtp2\n Status:Login Successful\n Message:User Granted Access Jul 27 01:48:31 nc0rtp223.us.nortel.com GWC: class_securit y.ver01 DOC=login STAT=success SRC.USR=rtp2 MID=MID_Nortel_MGC_auth.0011 Jul 27 01:48:39 nc0rtp223.us.nortel.com GWC: class_ security.ver01 CMD=forceout STAT=end SRC.USR=rtp2 SRC=47.140.162.112 DST.USR=rtp2 MID=MID_Nortel_MGC_auth.0003
FTP logs
This section contains GWC security FTP logs.
Login successful
Sample log: Jul 27 01:51:40 nc0rtp223.us.nortel.com GWC: class_securit y.ver01 CMD=connect STAT=start SRC=47.140.162.112 CONNECT=ftp_connect MID=MID_Nortel_MGC_auth.0001 Jul 27 01:51:45 nc0rtp223.us.nortel.com GWC: class_securit y.ver01 CMD=authenticate STAT=success SRC.USR=ftp MID=MID_Nortel_MGC_auth.0010
Login failure
Sample log: Jul 27 01:54:17 nc0rtp223.us.nortel.com GWC: class_securit y.ver01 CMD=connect STAT=start SRC=47.140.162.112 CONNECT=ftp_connect MID=MID_Nortel_MGC_auth.0001 Jul 27 01:54:20 nc0rtp223.us.nortel.com GWC: class_securit y.ver01 CMD=authenticate STAT=failure SRC.USR=ftp MID=MID_Nortel_MGC_auth.0010 Jul 27 01:54:20 nc0rtp223.us.nortel.com GWC: class_securit y.ver01 CMD=disconnect STAT=end SRC=47.140.162.112 MID=MID_Nortel_MGC_auth.0008
184
Logout
Sample log: Jul 27 01:52:20 nc0rtp223.us.nortel.com GWC: clas s_security.ver01 CMD=logout STAT=end SRC.USR=ftp SRC=47.140.162.112 MID=MID_Nortel_MGC_auth.0002
Serial connection logs

This section contains security serial connection logs.
Login successful
Sample log: Jun 21 17:09:57 [172.16.16.43.128.192] GWC class_security. ver01 CMD=connect STAT=start TTY=0 CONNECT=serial_connect MID=MID_Nortel_MGC_auth.0001^M Jun 21 17:10:18 india1sesm SuccessionLogin: UserId:ptm\n Status:Login Successful\n Message:User Granted Access^M Jun 21 17:10:18 [172.16.16.43.128.192] GWC class_security. ver01 DOC=login STAT=success SRC.USR=ptm MID=MID_Nortel_MGC_auth.0011
Forceout serial connection to GWC through telneting to GWC

Sample log: Jun 21 18:07:01 [172.16.16.43.128.192] GWC class_security. ver01 CMD=connect STAT=start SRC=164.164.27.43 CONNECT=telnet_connect MID=MID_Nortel_MGC_auth.0001 Jun 21 18:07:05 india1sesm SuccessionLogin: UserId:ptm\n Status:Login Successful\n Message:User Granted Access Jun 21 18:07:05 [172.16.16.43.128.192] GWC class_security. ver01 DOC=login STAT=success SRC.USR=ptm MID=MID_Nortel_MGC_auth.0011 Jun 21 18:07:13 [172.16.16.43.128.192] GWC class_security. ver01 CMD=forceout STAT=end SRC.USR=ptm TTY=0 DST.USR=ptm MID=MID_Nortel_MGC_auth.0003 Jun 21 18:07:22 [172.16.16.43.128.192] GWC class_security. ver01 CMD=connect STAT=start SRC=164.164.27.43 CONNECT=telnet_connect MID=MID_Nortel_MGC_auth.0001 Jun 21 18:07:38 india1sesm SuccessionLogin: UserId:ptm\n Status:Login Successful\n Message:User Granted Access Jun 21 18:07:49 [172.16.16.43.128.192] GWC class_security. ver01 DOC=login STAT=success SRC.USR=ptmMID=MID_Nortel_MG C_auth.0011
Serial connection logs
185
Jun 21 18:07:58 [172.16.16.26.128.137] class_security.ve r01 CMD=logout STAT=end SRC.USR= TTY=0MID=MID_Nortel_MG C_auth.0006
Logout
Sample log: Jun 21 17:10:47 [172.16.16.43.128.192] GWC class_security. ver01 CMD=logout STAT=end SRC.USR=ptm TTY=0 MID=MID_Nortel_MGC_auth.0002^M Jun 21 17:10:47 [172.16.16.43.128.192] GWC class_security. ver01 CMD=disconnect STAT=end TTY=0 MID=MID_Nortel_MGC_auth.0008^M
186
Carrier VoIP
Nortel Gateway Controller Fault Management

Release: GWC120 Publication: NN10202-911 Document revision: 11.03 Document release date: 14 October 2009 Copyright 2009 Nortel Networks. All Rights Reserved. While the information in this document is believed to be accurate and reliable, except as otherwise expressly agreed to in writing NORTEL PROVIDES THIS DOCUMENT "AS IS" WITHOUT WARRANTY OR CONDITION OF ANY KIND, EITHER EXPRESS OR IMPLIED. The information and/or products described in this document are subject to change without notice.
Nortel, Nortel Networks, the Nortel logo, and the Globemark are trademarks of Nortel Networks. All other trademarks are the property of their respective owners. To provide feedback or to report a problem in this document, go to www.nortel.com/documentfeedback. www.nortel.com

GWC Fault Management - (I) CVM12

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

GWC Fault Management - (I) CVM12

Uploaded by

Copyright:

Available Formats

Carrier VoIP

Nortel Gateway Controller Fault Management

All other trademarks are the property of their respective owners.

Fault management fundamentals

Fault monitoring and troubleshooting procedures

Fault clearing procedures

Alarms and logs reference

New in this release

Features (page 5) Other changes (page 5)

There were no feature changes made to this document.

6 New in this release

Fault management fundamentals

Fault management strategy for GWC

10 Fault management fundamentals

Alarm severity codes

Fault management strategy for GWC

12 Fault management fundamentals

Alarm information is sent to:

Tools and utilities overview

Logs and alarms associated with IPSec

Tools and utilities overview

14 Fault management fundamentals

Integrated Element Management System

"Launching GWC Manager" "Launching SAM21 Manager"

Use of ping and traceroute

Fault management in a DQoS network

Fault management in a DQoS network

16 Fault management fundamentals

DQoS COPS alarm description

GWC card auto-recovery and boot auditing

Troubleshooting DQoS/COPS connection failures

GWC card auto-recovery and boot auditing

1. The Motorola firmware on the GWC card performs a network autoboot

2. If the network autoboot fails, the SAM21 shelf controller performs

18 Fault management fundamentals

20 Fault management fundamentals

Replace these air filters once every 10,000 hours (approximately

SPD/LNK (green/yellow; MCPN905 only) - Ethernet link speed and

ACT (green; MCPN905 only) - Ethernet link activity; lights when

CPU (green) - CPU activity; lights when the cards processor is

BFL (yellow) - Board failure; lights when a system failure occurs

Hot swap status (blue; in handles of MCPN905 card) - Lights when

22 Fault management fundamentals Figure 3 Details: Motorola N905 NSS board

Figure 4 Details: Motorola N750 NSS board

24 Fault management fundamentals

Data integrity audits

Restrictions and limitations

Data integrity audits

CS 2000 data integrity audit

On the CS 2000 Management Tools server:

the network instance and fabric type contained in the BEARNETS

Line data integrity audit

26 Fault management fundamentals

Data integrity audits

Trunk data integrity audit

SERVRINV TRKMEM LTMAP TRKSGRP

V5.2 data integrity audit

28 Fault management fundamentals

Fault monitoring and troubleshooting procedures

30 Fault monitoring and troubleshooting procedures

Viewing GWC service alarm history

At the CS 2000 GWC Manager client