NOMD PROBLEM INVESTIGATION 2020

PROBLEM INVESTIGATION REPORT

MPT072 MSS8 KA-PKM

Customer : Telekom Malaysia

Project Name : IP MICROWAVE RADIO

AR No : SRF CTAC SR 22/2020

Site Name : MPT072 MSS8 KA-PKM

Status : CLOSE

Start Time/Date : 28/5/2020 10:00:00 AM

Close Time/Date : 28/5/2020 02:30:00 PM

Prepared By : DIANA ROS

Verified By : SHAHIRAN

NOMD Reference. : SRTSC-2020/18

Table of Contents
 NOMD PROBLEM INVESTIGATION 2020

1.0 Troubleshooting record Table

2.0 Problem Description

3.0 Investigation and Analysis

4.0 Root Cause & Solution

5.0 Recommendations
 NOMD PROBLEM INVESTIGATION 2020

1.0 Troubleshooting Record Table

Name of the Product: 9500 MPR (MSS8)

Date/Time arrived: 28 May 2020 Time of solution: 28 May 2020

Person on duty: Mohd Shaiful Azli & Mawardi Site Name: MPT072 MSS8 KA-PKM

Fault type:
Hardware fault
Power fault
Software fault
Fiber interface fault
Configuration data error
Other faults (Please describe) – NMS problem

Fault source:
■User complaint
■Alarm system
Found during routine maintenance
Other sources

Description of Fault:

1. Customer report 2Mbps data down : Maybank Pulau Ketam (PKM)

2.0 Problem Description

 NOMD PROBLEM INVESTIGATION 2020

Telekom Malaysia request for Engineering Support for our IP ALU network:

LINK ID: MPT072 MSS8 KA-PKM

Location: TM Kapar Exchange
Fault Description: Customer report 2Mbps data down: Maybank Pulau Ketam (PKM)

Figure 2.1 – MPT033 MSS1 SJN-SSJN

3.0 Investigation and Analysis

Chronology of events:

Date/time Events

27/05/202 - Received email AR from FaultMgmt at 2.57 pm.

0 - Info from CNZ on 27/05/2020 TM’s customer, Maybank Pulau Ketam, their data
(2Mbps) is failure.
- TM CNZ try to solve first on 27/05/2020 by run E1 test at Pulau Ketam.
- SR try assist by remote but unfortunately unable due to NMS problem.
- Request by CNZ to go to the site on 28/05/2020 appointment at 10:00 am. Meet at
TM Kapar.
- Request PTW from TM to access the site. PTW approved at 5.39 pm.

28/05/202 - Arrived at TM Kapar at 9.30 am and waiting TM CNZ to arrive until 10.00 am.
0 - Enter equipment location room and check the NMS problem first.
- Identified the NMS’s switch location and the port.
- Check the NMS cable and port. Verified all OK.
- Try to call NMS to access via VPN and ping the IP switch but not successful.
- Call TM GIT to investigate.
- Access NETO to identify the alarm on E1 port/card.
- Check E1 xconnect and port configuration.
- Found E1 card power failure show in the system.
- Request CNZ to do physical loopback at TM Kapar and run test gear at PKM.
- All setup done and waiting CNZ team to arrive at TM PKM.
- CNZ team arrived at PKM on 10.45 am.
 NOMD PROBLEM INVESTIGATION 2020

- First thing to do hard reset E1 card at PKM before run test.

- After hard reset, E1 card at PKM working normal and run test.
- Run test at port 1, loopback at DDF TM Kapar, result was OK.
- Run test and loopback at DDF MSS8 panel, result not OK.
- Run test and do software loopback, result not OK.
- Request by CNZ change to port 5.
- Try to do xconnect and enable port, result not OK.
- Troubleshoot the problem, found some error on configuration data which is cannot
overwrite/delete the old data.
- Run test at port 5, justified port was OK and cnz request just continue using port 5
instead of port 1.
- Put thru circuit (Maybank) and waiting them to justify.
- Time Restored at 2:30 pm.

Initial check via NETO at site, target to bring up NMS access via VPN/SAM to be OK first. The
purposed is to make monitoring system via SAM/NMS back to normal again.

Figure 3.1 – Alarm detected

On site troubleshooting, firstly identified switch and port location for the NMS. Then we checked the
NMS cable and port ok or not. After verified all OK, try call NMS to login via VPN/SAM. Result not
OK. Then we call TM GIT, Ms. Raja Emy to investigate and told her to check the routing for the
NMS. Then she replied as figure below.

Figure 3.2 – NMS problem verification

After that try to call NMS to ping and login via VPN/SAM. The result show below.
 NOMD PROBLEM INVESTIGATION 2020

Figure 3.3 – NMS ping site from via VPN/SAM

Figure 3.4 – MPT072 MSS8 KA-PKM (after)

After that, we troubleshoot E1 port 1 problem. Target to bring up 2Mbps data service for Maybank
Pulau Ketam, the main AR issue.

Figure 3.5 – MPT072 MSS8 at TM PKM (before)

 NOMD PROBLEM INVESTIGATION 2020

Firstly, need to hard reset E1 card at TM PKM because the card seems like not power up (abnormal)
and hanged.

LED not active

Figure 3.6 – E1 card no power at TM PKM (before)

Figure 3.7 – Alarm for E1 card hard reset

After hard reset E1 card at TM PKM, the E1 card back to normal.

Figure 3.8 – E1 card power up at TM PKM look like (after)

We try to run test gear at TM PKM and do physical loopback at DDF in TM Kapar. Run port is port
1. Verified test was OK after run it at DDF. After that, verify by physical loopback at DDF
Equipment panel.

Port 1 Tx & Rx

Figure 3.9 – E1 DDF Equipment panel

Run test and result was not OK. Request by CNZ to run at port 5 because port 5 is available. Config
xconnect for port 5.The reason on delaying config port 5 and xconnect is because of the error
message by the database cannot overwrite new configuration data.
 NOMD PROBLEM INVESTIGATION 2020

Figure 3.10 – Configuration error when trying to config port 5 and xconnect

We try to hard reset the E1 card both sites and then NE software reset.

Figure 3.11 – Alarm for software reset NE (at TM KA)

Figure 3.12 – Alarm for software reset NE (At TM PKM)

After doing that, port 5 can be config and its xconnect can be done.
NOMD PROBLEM INVESTIGATION 2020

Figure 3.13 – port 5 config (at TM KA)

Figure 3.14 – xconnect config (at TM KA)

NOMD PROBLEM INVESTIGATION 2020

Figure 3.15 – port 5 config (at TM PKM)

Figure 3.16 – xconnect config (at TM PKM)

 NOMD PROBLEM INVESTIGATION 2020

Summary Noted:-

1. Actual issue including:

a. Why can’t using port 1 for restoration?
* Request by CNZ to use port 5 after restoration after they confirm with Maybank the data ONLINE.
b. Why NE offline in NMS?
* The answer is in figure 3.2 above.
2. Detail solution including:
a. Why need to restart several time both NE card E1 for restoration?
*1st reset – for restoring E1 card to be online
*2nd reset – for reloading new configuration. (Noticed that, sometimes it need to reset several time
because the card seems offline for quite long time.)
*NE software reset – to overwrite new data configuration to DB and refresh NE for both sites. If we not do
that, any new configuration from the system will not overwrite to DB.

Figure 3.17 – Network Diagram

Figure 3.18 – Alarm (after)

 NOMD PROBLEM INVESTIGATION 2020

4.0 Root Cause & Solution

4.01 NMS Problem

Root caused: Due to Leased line Metro.E migration at TM Kapar but TM PKM still using ip
Metro.E

Solution: Checked with TM GIT the IP routing and configuration after on site verification was
OK.

4.02 E1 (2Mbps) port problem

Root caused: E1 card offline and software problem.

Solution: Hard reset E1 card and Software reset NE. Re-config xconnect and E1 port. Run test for
confirmation.

5.0 Recommendations
The service card to be offline for a long time will lead to card abnormal function. PM (Preventive
Maintenance) will make sure all the system to be working as normal condition.

Any new updating for NMS routing or config need to be inform to avoid any NMS network offline.