Tier2 Handbook

Purpose Of Handbook

This handbook has been prepared to help new comers to understand the
responsibilities of Tier2 team which is one of NOC TE Data teams.
Also after reading the following papers you should be aware by team policies
and you will be able to start to concentrate on learning the technical part
which will help you to solve our technical problems successfully.

Team Responsibilities

1 - Working on All Tickets (OTS) at NOC pool also following on all tickets
pending with other teams.

2- Any Case has hardware failure we are the responsible for preparing new
one and sending it.

3- Opening cases with our vendors (Cisco & juniper) in case of crisis.

4- Trying to find solutions in case of crisis causing impact to reduce it.

5- Covering Network maintenance team after 10 PM every day.

6- Working on MSAN cabinets in case of logical or configuration problems

and following with TE in case of hardware problems.
Type Of Problems

problem

Total partial service

None degradation managemet

Router Switch interface Dslam card Temp.

down packet loss

Dslam Cabinet

1- Problem classified as a total down when there is (Router, DSLAM,

Switch or cabinet) totally down. This cases opened by Tier1 at NOC
pool. NOC engineer should assign it to network maintenance pool and
to his Name to be the responsible for it.
2- Problem classified as Partial if there is interface down on the router or
DSLAM card down (part of device impacted).
If the problem related to hardware failure that's Tier2 engineer
responsibility so the ticket opened at NOC pool.
If the problem related to transmission problem, ticket opened by tier1
and assigned to network maintenance team. Tier 2 engineer should
follow.
3- Problem classified as none when the device and it's all parts up but
there is another problem such as Temperature cases and CPU or
memory problems.
4- Problem classified as service degradation when there is a packet loss
problem on a device or on a transmission link between 2 devices or
POPs.
5- Problem classified as management when the device is up normally
and no impact on the service but we can't manage the device.
Tickets Flow

You can check the below flowchart which illustrate the OTS flow,
beginning with it's opening till problem recovery and closing it.

Note :

NM: Network maintenance Team

H/W: Hardware problem

Team Public Folder And Documentations

You can open the below link to access Tier 2 documentations on NOC public
folder

\\213.158.174.6\noc\Tier2 Documents

Username: noc
Password: TEDatan0c
Escalation Matrix
1- MSAN

In cases of down outdoor MSAN cabinet we or (maintenance at their working

hours) sending mail to Level 1 of escalation matrix, if there is no reply we can
escalate to level 2 and level 3.

2- Transmission O&M

In case we have device down at certain POP we call TE shift , if no answer for
30 min we will call the next level of escalation (head of section), if no answer
after another 30 min call the next level (head of department) after one hour if
no answer we can call the last level (General Manager).Note: All escalation
matrix sheets available on shared folder.
path: \\213.158.174.6\noc\Tier2 Documents\Escalation matrix

3- Opening CO.

After working hours if we need to access CO for any reason, we will send mail
with all details of the CO we need to access, also all details of the technician
to the following mails (cmc@te.eg & CMC-Frontdesk@te.eg ),You can check
below example:
In case CMC replied with no security, then we will escalate to RAMSIS
Maintenance shift and he will escalate and check with CO. Manager and
area manager to open CO.

Regarding Test Room of CO.: Maintenance team is the responsible to open it

during their working hours, after that RAMSIS shift will check.

Handover Criteria
- At the end of every working day, team should handover with all cases
to the next shift (morning or night) to continue working.
- At the end of every shift, we should prepare Pending/Critical mail, this
mail should contain handover of all pending cases, attach critical
mails, attach weekly shift and updated IP local pool sheet.

Good Handover will help other team to working on the pending cases with
high performance as it will save time to understand each case.

Store And Lab

NOC Store:

It’s the room which contains all hardware spare parts we can use to replace
faulty one.
Each shelf containing a kind of hardware.
Dismiss Hardware:

Every time we dismiss hardware from store after normal working hours we
should send mail to store keeper to inform him with the kind of hardware and
its related OTS.
Retrieve Hardware:

Every time we retrieve hardware to store after normal working hours we

should send mail to store keeper with the hardware and OTS also clarification
if the hardware faulty or working fine.

NOC Lab:

It’s the room which contains powered DSLAMS shelves and DC powered
routers such as MX104, we using it to test and configure hardware before
sending it.
Teams We Are Following With

1- Access operation

Following with them when there is old activated DSLAM or cabinet and facing
problem such as (missing configuration, advice before changing hardware,
unable to obtain IP with no clear reason).

2- Core operation

Following up with them if we need advice before changing router hardware,

also when there is unknown behavior on one of routers and also if we need to
open Case with vendors Cisco & juniper.

3- Commissioning

Following with them when there is newly activated cabinet and facing
problem such as (missing configuration, unable to obtain ip with no clear
reason). Or to confirm when DSLAM converted to MSAN cabinet.

4- Access engineering

Check with them if there is unbalanced traffic on DSLAM or if there is

congestion on DSLAM.

5- Core engineering

Check with them if there is limitation on core routers, also its responsibility to
add and remove interfaces of transmission on routers.

6- Capacity planning

Check with them pop congestion , if you need to upgrade transmission , also
if there is transmission between pops exist on network but has no data on
NOMS we then check with CP to confirm if this transmission removed.
Brief About Our Tools:

1- OTS (Outage Tracking System)

We are using this tool to handle our tickets and to following us with
other teams using this tool, we can use it to open new ticket, closing
tickets and update it.
We can update OTS as below

We can close it as below

2- NOMS

We are using this tool to know any details about any device in the
network (Routers, DSLAMs, MSAN cabinets and etc.) also to know info
about transmission between pops.
As per below view we use search Devices to display certain device
details or all devices at our network
MSAN search Devices to display details about certain MSAN cabinet or
to display all MSAN cabinets at our network.

Also from the below view we can use search by POP to display all
transmission and devices in this pop.
 Also we can use List ALL POPs to display all POPs names by Arabic
language

3- NST

We are using this tool to view details about certain ADSL account such
as (IP address, PPPOE session logs and account profile). We are using it
to check logical ADSL cases such as unable to obtain IP and NAS port
ID problems.
4- MRTG

URL: http://network.tedata.net

We are using the A/M URL to access tool; we are using it to view
graphs of network devices to check utilization of devices uplinks and
interfaces, also transmission and international links.
As below graph of one of international link.
5- Matrix

We are using this tool to get all details about ADSL account (it's DSLAM
or cabinet and BRAS, pop name). Also to know the status of the ADSL
account. Also we can use it for more functions such as checking all
account on DSLAM or cabinet.
Basically it's related to ADSL service
We can use Select POP and Cards to display all ADSL customers at
certain DSLAM and certain Card.

We then choose the pop name and DSLAM number; also we can
select cards range we need to display.

BY: Mohamed Nada