Fabric drops

Added by Yannick Le Teigner, last edited by Yannick Le Teigner on Apr 07, 2011 (view change)

This counter indicate some drops sending the packets across the fabric.

Collect the 'show class-of-service fabric statistics' command, and see if any FPC has drops towards
another FPC.
timeden@re0-bx01.lax04> show class-of-service fabric statistics
[...SNIP...]
Destination FPC Index: 8, Source FPC Index: 7
Total statistics: High priority Low priority
Packets: 289633053685 735213313900
Bytes : 93612800544377 554762954469973
Pps : 535679 958963
Bps : 181043816 541669832
Tx statistics: High priority Low priority
Packets: 289633053685 735200362884
Bytes : 93612800544377 554750982278504
Pps : 535679 958963
Bps : 181043816 541669832
Drop statistics: High priority Low priority
Packets: 0 12951016 <<<
Bytes : 0 11972191469 <<<
Pps : 0 0
Bps : 0 0

Destination FPC Index: 8, Source FPC Index: 8

Total statistics: High priority Low priority
Packets: 333180208765 832613112002
Bytes : 108152519187544 591250925225834
Pps : 638957 1137250
Bps : 214903213 631755110
Tx statistics: High priority Low priority
Packets: 333180208765 832587861203
Bytes : 108152519187544 591227850078720
 Pps : 638957 1137250
Bps : 214903213 631755110
Drop statistics: High priority Low priority
Packets: 0 25250799 <<<
Bytes : 0 23075147114 <<<
Pps : 0 0
Bps : 0 0

In the above output, you can see drops from FPC7 and FPC8 to FPC8. This command doesn't tell you
which PFE is actually having the drops. So you'll need to dig deeper to find out the problematic set of
interfaces. And depending on the type of FPC, you'll have to use different commands.

Trinity
The bandwidth for the Trinity based PFE on an MX960 is about ~29G. So it's possible to overload the
fabric before the interfaces.
If you suspect fabric overloading from the PFE, You first need to identify each stream that is going from
the FPCs dropping packets (as identified above).
There's one fabric stream for each MqChip going to every other PFE in the chassis.
There are 4 stream IDs per FPC, so FPC 7 will have stream IDs 28, 29, 30 and 31, whereas FPC 8 will
have stream IDs 32, 33, 34 and 35.

Below command also shows the mapping:

NPC8(sting-re0 vty)# show mqchip 0 stream all

Input Streams enabled (F=FAB, W=WAN, D=DROP, L=LPBK, H=HOST)

-------------------------------------------------------------
0.. 63: ----FF--FFFF------------FFFFFFFF FFFF--------------------------
--
64.. 127: -------------------------------- ------------------------------
--
128.. 191: ----FF--FFFF------------FFFFFFFF FFFF--------------------------
--
192.. 255: -------------------------------- ------------------------------
--
256.. 319: -------------------------------- ------------------------------
--
320.. 383: -------------------------------- ------------------------------
--
384.. 447: -------------------------------- ------------------------------
--
448.. 511: -------------------------------- ------------------------------
--
 512.. 575: -------------------------------- ------------------------------
--
576.. 639: -------------------------------- ------------------------------
--
640.. 703: -------------------------------- ------------------------------
--
704.. 767: -------------------------------- ------------------------------
--
768.. 831: -------------------------------- ------------------------------
--
832.. 895: -------------------------------- ------------------------------
--
896.. 959: -------------------------------- ------------------------------
--
960..1023: -------------------------------- ------------------------------
--
1024..1087: -WWW-WWW-WWW-WWWD--------------- ------------------------------
--
1088..1151: -------------------------------- --------------------
LLLLLLLLLLLH

Output Streams enabled (F=FAB, W=WAN, D=DROP, L=LPBK, H=HOST)

-------------------------------------------------------------
0.. 63: ----FF--FFFF------------FFFFFFFF FFFF--------------------------
--
64.. 127: -------------------------------- ------------------------------
--
128.. 191: ----FF--FFFF------------FFFFFFFF FFFF--------------------------
--
192.. 255: -------------------------------- ------------------------------
--
256.. 319: -------------------------------- ------------------------------
--
320.. 383: -------------------------------- ------------------------------
--
384.. 447: -------------------------------- ------------------------------
--
448.. 511: -------------------------------- ------------------------------
--
512.. 575: -------------------------------- ------------------------------
--
 576.. 639: -------------------------------- ------------------------------
--
640.. 703: -------------------------------- ------------------------------
--
704.. 767: -------------------------------- ------------------------------
--
768.. 831: -------------------------------- ------------------------------
--
832.. 895: -------------------------------- ------------------------------
--
896.. 959: -------------------------------- ------------------------------
--
960..1023: -------------------------------- ------------------------------
--
1024..1087: WWWW---------------------------- ------------------------------
--
1088..1151: -------------------------------- --------------------
LLLLLLLLLLLH

Each 'F' letter match a fabric stream.

Stream IDs below 128 are for low priority streams, above are for high priority streams.
Once you have the IDs, you can run below command to see if this specific PFE is dropping packet:
NPC7(re0-bx01.lax04 vty)# show mqchip 1 counters output stream 34

DSTAT phy_stream 34 Queue Counters:

Aggregated Queue Stats QSYS 1 Qs 34..34 Colors 0..3:

Counter Packets Pkt Rate Bytes Byte

Rate
-------------------- ---------------- ------------ ---------------- -------
-----
Forwarded (NoRule) 0 0 0
0
Forwarded (Rule) 492678393325 691904 402280015848109
411590958
Dropped (WRED) 0 0 0
0
Dropped (TAIL) 12951016 0 11972191469
0 <<<
Dropped (Force) 0 0 0
0
 Dropped (Error) 0 0 0
0

FO Counters:

Stream Mask Match Packets Pkt Rate Bytes Byte

Rate Cells Cell Rate
-------------------- ---------------- ------------ ---------------- -------
----- ---------------- ------------
FO0 0x000 0x000 1827657487930 1769749 1230568933916681
932674965 19902024509816 15305377
FO0 0x000 0x000 1827657487949 1769751 1230568933926896
932678952 19902024509986 15305439
FO1 0x000 0x000 0 0 0
0 0 0
FO1 0x000 0x000 0 0 0
0 0 0

Above output shows PFE7/1 (mqchip1) to FPE 8/3 (stream 34).

This is most likely caused by excessive traffic going to the destination PFE.

You can find additional information here .