Electrical Subrack Offline Issue with

Troubleshooting Steps
Created:  Feb 9, 2022 18:41:31Latest reply: Feb 19, 2022 17:08:06 909 15 8 0 1
View the author 1#

Hello Community!

This post describes an electrical subrack offline issue with troubleshooting steps. Please have
a read below to find more details on the subject.

ISSUE DESCRIPTION

A heavy outage was recently observed in one of our regional networks due to an

offline issue of the electrical subrack at Site-S while the optical subrack was up
keeping the ring intact at the optical layer. OSN 1800V and PTN 980 lost their
management and caused service interruption. The outage was observed on multiple
sites (2G+3G+4G).

Network topology on NCE

ANALYSIS

Checked the alarms and operation records generated during the time when the alarm
persisted. 

 Product: Huawei OSN 1800V.

 NMS: iMaster NCE-T.

TROUBLESHOOTING STEPS

Checked the alarms and operation records generated during the time when the outage
occurred. 

We started our troubleshooting with NCE-T. Upon checking, the NMS Optical NE

was online and the Electrical NE was offline. Form the alarms of Optical and
Electrical NE at NMS.
Power_Fail and FAN_Fail alarms were reported on the Optical NE, while the
Electrical NE only reported the REM-SF and OPU2_CSF alarms and got offline.

Checked with a team who ware already on site and shared the snap of the OSN
1800V equipment showing all boards lights off. FAN board LED was red.

After plugging out the FAN, the FANs inside the board were not working. And
physically all boards were very hot.
Upon these findings, we immediately directed the customer to arrange those boards
so that the faulty boards could be replaced and outages could be fixed:

 2 x Z5UXCMS;

 2 x F5TTA;

 4 x UNQ2;

 1 x F5FAN.

The team arranged the board from Warehouse Spare and replaced the FAN and a
UXCMS board. The remaining old boards started working and the affected sites get
restored.

Then we arranged the OSN 9800 FANs the next day and replaced the faulty FANs
onsite.

Onsite rectifier alarms

Battery Reversely Connection alarms occurred at the rectifier end. Due to this, a
DC surge occurred and caused a high power usage cards to get faulty on the OSN
end.

FURTHER ANALYSIS

OPU2_CSF

The OPU_CSF alarm indicates that the client-side signal fails and is generated when
the client-side signal of the remote end fails. Usually, when this alarm occurs, it is in
pair with the client-side of the remote end.

REM-SF

This one also indicates a client-side signal fail and is generated when the client-side
signal of the remote end fails.

ROOT CAUSE

Due to an abnormal DC power, the onsite FANs of OSN 1800V and OSN 9800 UPS
get faulty and a site outage occurres. After replacing the faulty boards onsite, all sites
get restored.

RECCOMENDATIONS
  A steady power supply with redundancy for transmission equipment is
essential for any network. Proper power connections and a DC Surge arrestor
should be installed for protection.

 Preventive maintenance (PM) precautions and steps are required to avoid

accidents or equipment failures from occurring before they happen. 

 To ensure good heat dissipation and ventilation for the system and to prevent
the accumulation of dust on an air filter, it is needed to clean the air filter
regularly.

A more detailed discussion can be found here: Transmission Network Reliability

Assurance Plan.

Please feel free to leave a message and exchange knowledge in the comment area.
Thank you!

Visit my personal Author collection, which contains the most valuable articles written
around the areas of IP core/Datacom, Cloud, access (FTTH), transmission and
emerging technologies.
 
With thanks,

Bashir Ahmed Zeeshan