Professional Documents
Culture Documents
Troubleshooting Updates, RMA Process & M120 Overview: 2009 Q1 Jason Abercrombie Resident Engineer
Troubleshooting Updates, RMA Process & M120 Overview: 2009 Q1 Jason Abercrombie Resident Engineer
Troubleshooting Updates, RMA Process & M120 Overview: 2009 Q1 Jason Abercrombie Resident Engineer
RMA Process
&
M120 Overview
2009 Q1
Jason Abercrombie
Resident Engineer
Agenda
Troubleshooting Updates
LCHIP
RXXG
RMA Process
M120 Overview
Craft Interface
Cooling
Flexible PIC Concentrator (FPC) and Compact FPC (cFPC)
Control Board
Routing Engine
Forwarding Engine Board
Power Distribution
LCHIP Update
LCHIP errors can be expected if...
- An interface is flapping interface on the FPC experiencing the LCHIP errors
- The FPC has bounced (should also see HDRF CRC errors in DESRD)
Troubleshooting Review:
Error-checking on M320 and T640 occurs once, on egress FPCs LCHIP
The error can occur anywhere along the data path
1 or more ingress FPCs Fabric (SIBs) Egress FPC
Goal: Find the faulty SIB or FPC
Work from least impact to most impact
1. Cycle the SIBs one by one, checking if errors subside
2. Cycle the FPCs one by one, checking if errors subside
RXXG Update
Jan 13 06:35:58 2009
cer-pcor-03 fpc7 .pm3393.7.1. RXXG: %PFE-3: Packet exceeds the maximum frame size 1526
These messages appear when LINE_ERRI bit is set on PMC pm3393 MAC chip.
LINE_ERRI
The LINE_ERRI bit is set when a Line Interface error is detected. Failure modes are (1)
Breakdown in alternating SOP-EOP sequence (2) Invalid byte(s) between SOP and EOP
not part of a completely invalid word. (3) Reception of frames less than 14 bytes.
PR 80772 RXXG messages are reported for invalid frames
PR 100022 RXXG messages logged too often
PR (Internal) XENPAK triggers confusing messages as a result of packet loss
RXXG messages are a symptom of a real failure that has occurred at the MAC layer.
RXXG Update
Cases opened in 2008
Case Number and Routers
Solution
2008-0416-0238 atl-pcor-02
2008-0813-0055 dca-pcor-01
2008-1025-0089 phx-core-02
2008-1110-0731 dca-pcor-2
2008-1117-0452 slkc-agw1
2008-1121-0237 cer-edge-12
2008-1210-0704 cls-core-02
2008-1211-0683 cer-pcor-01
2008-1211-0692 dca-core-01
2008-1215-0316 jfk-pcor-01
JTAC suggested to bounce PIC and test transport. Closed after inactivity
2008-1224-0224 dca-pcor-01
RXXG Update
792 Xenpak xcvrs in the network
42 (5.3%) reported >1 errors in 2009
Including only:
RXXG / line interface errors
RXXG / exceeds maximum frame size
Not including RXOAM messages (not errors)
41 of the 42 (97.6%) xcvrs have P/N 740-013170
766 of all 792 (96.7%) xcvrs have this P/N
RXXG Update
XENPAK Xcvrs reporting errors in 2009
C715TF012
T07D05561
T07K46483
T06F90384
T07D05563
T07K46505
T07C93701
T07D05572
T07K46512
T07C94487
T07D05574
T07M71281
T07C96411
T07D05580
T07M71290
T07C96431
T07E25431
T07M71325
T07D04670
T07E32650
T07M71418
T07D04676
T07F53170
T07M71441
T07D04749
T07G74116
T07M71443
T07D04761
T07J04552
T07M71527
T07D04766
T07J04761
T07M71574
T07D04780
T07K39450
T07M71624
T07D04821
T07K46381
T08A15335
T07D04824
T07K46440
T08B21207
RXXG Update
Three causes for these errors
- Bad transmitting device
- Line impairment
- Problem at receiving side
Local Side
- Collect Data
Frequency of Errors
Incrementing Counters (CRC/Align, Jabber frames, etc.)
- Clean Fiber
- Replace PIC (PIC replacement fixes the problem in most cases)
Remote Side Testing As Necessary
Line/Transport Testing As Necessary
Agenda
Troubleshooting Updates
LCHIP
RXXG
RMA Process
M120 Overview
Craft Interface
Cooling
Flexible PIC Concentrator (FPC) and Compact FPC (cFPC)
Control Board
Routing Engine
Forwarding Engine Board
Power Distribution
10
Agenda
Troubleshooting Updates
LCHIP
RXXG
RMA Process
M120 Overview
Craft Interface
Cooling
Flexible PIC Concentrator (FPC) and Compact FPC (cFPC)
Control Board
Routing Engine
Forwarding Engine Board
Power Distribution
11
Components
Hardware inventory:
Item
Chassis
Midplane
REV 04
FPM Board
Serial number
Description
JN109326EAEA
M120
710-018041
RC2032
M120 Midplane
REV 06
710-011407
DM3006
FPM Display
REV 02
710-011405
RH1106
FPM CIP
REV 05
710-011410
RH1117
PEM 0
Rev 10
740-011935
TL53577
PEM 1
Rev 10
740-011935
TL53737
740-014082
9009004730
RE-A-2000
740-014082
9009004240
RE-A-2000
CB 0
REV 09
710-011403
DL2561
CB 1
REV 09
710-011403
DM5467
FPC 2
REV 03
710-015837
DM3717
REV 07
750-010618
DM5012
PIC 0
Part number
Xcvr 0
REV 01
740-011613
AM0813S91DA
Xcvr 1
REV 01
740-011614
84S495H11736
SFP-LX
PIC 1
REV 25
750-001901
WP3316
PIC 2
REV 12
750-009066
WM1052
REV 01
740-011786
768002D00120
SFP-IR
REV 04
710-015838
DM0335
REV 03
710-015837
DM3720
REV 07
750-010618
DM2489
Xcvr 0
Board B
FPC 3
PIC 0
Xcvr 0
REV 01
740-011613
PD50TDN
Xcvr 1
REV 01
740-011614
84S495H11733
SFP-LX
REV 12
750-009066
WM1058
Version
PIC 2
Xcvr 0
Board B
FPC 4
REV 01
740-011786
798002D00402
SFP-IR
REV 04
710-015838
DM0362
REV 03
710-015835
RH1939
PIC 0
REV 22
750-005634
DP0143
PIC 1
REV 13
750-003034
RG6352
Board B
REV 03
710-017980
RH1208
FEB 0
REV 05
710-015795
RG9019
M120 FEB
FEB 1
REV 05
710-015795
DN4401
M120 FEB
FEB 2
REV 05
710-015795
DK1553
M120 FEB
FEB 3
REV 05
710-015795
DN4431
M120 FEB
FEB 4
REV 05
710-015795
DN4356
M120 FEB
FEB 5
REV 05
710-015795
DK1518
M120 FEB
Fan Tray 0
Fan Tray 1
Fan Tray 2
Fan Tray 3
12
Front Components
Craft Interface
Front Top Fan Tray
cFPCs
PIC
FPC
Front Bottom Fan
Tray
Copyright 2008 Juniper Networks, Inc.
13
Craft Interface
Yellow and Red
Alarm LEDs, and
Alarm Cut-off button
External clock ports
Alarm relay contacts
PEM LEDs
RE0 ports
RE1 ports
RE and CB
LEDs
FEB LEDs
14
Craft Interface
Yellow/Red Alarms and Cut-off
Cut-off Deactivates RED and YELLOW alarms, and tests all LEDs
when pressed and held
EXT CLOCK Ports A and B accept two RJ-45s for clock input with T1 or
E1 reference clocks
RE Ports
15
Craft Interface
Yellow/Red Alarms and Cut-off
FPC LEDs and Buttons
Status
Steady GRN
Blinking GRN
Steady RED
Steady GRN
Steady RED
Steady GRN
Steady GRN
Steady RED
Steady GRN
Blinking GRN
Steady RED
RE is master
RE is functioning
RE has failed
CB is active
CB is transitioning online/offline
CB has failed
Steady GRN
Blinking GRN
Steady GRN
Steady RED
FEB is active
PEM LEDs
Status
RE/CB LEDs
RE Master
RE Status
CB Status
FEB LEDs
Active
Status
16
Cooling
Fan
Fan
Fan
Fan
Tray
Tray
Tray
Tray
0
1
2
3
17
18
19
Rear Components
Rear Top Fan Tray
CB0
RE0
FEBs
PEMs
Rear Bottom Fan
Tray
Copyright 2008 Juniper Networks, Inc.
20
Control Board
CB works with RE to provide control and monitoring functions
Determine RE mastership
Control power and reset for the other router components
Connect FEBs and FPCs
Monitor and control fan speed
Monitor system status
Switch fabric
Redundant configuration
If two CBs are installed, one functions as the master CB and the other as
its backup. If the master fails or is removed, the backup restarts and
becomes the master.
CBs are hot-pluggable. If a CB fails and switches mastership to the
redundant CB, the Routing Engine mastership switches as well.
21
Routing Engine
Boot Order
USB device
Internal flash disk
HDD
LAN
22
23
Crossbar switch
Provides connection between FEB WAN links and
FPC WAN links
24
Power Distribution
Non-redundant
PEM 1
PEM 0
Redundant
Same position
whether AC or DC
25
Summary / Questions?
Troubleshooting Updates
LCHIP
RXXG
RMA Process
M120 Overview
Craft Interface
Cooling
Flexible PIC Concentrator (FPC) and Compact FPC (cFPC)
Control Board
Routing Engine
Forwarding Engine Board
Power Distribution
26
27