Big Buddha

WS-3548-XL

11/11/99
Presenter - Fred Schindler

Cisco Systems Confidential FRS B2

1
 What IS Big Buddha?

• Ethernet Switch
48 Ports, 10Mbit/100Mbit
2 GBIC Ports, 1000Mb

• Managed i.e. MicroProcessor based

• Cisco IOS Software - A lot of Networking
features in a low-cost product.
• 1rU chassis

Cisco Systems Confidential FRS B2

2
Buddha (B2) vs Tahiti Differences
• 48 10/100 ports
vs 24
• Manhattan integrated PHY & Bridge
vs Del Mar and separate PHY
• Avalon-64
vs Avalon-40
• Avalon Data Buffers 32Mb SGRAM-7
vs 8Mb SGRAM-8

Cisco Systems Confidential FRS B2

3
Buddha (B2) vs Tahiti Differences

• 50 MHz PHY clocks

vs 25MHz
• 3.3V main supply voltage
vs 5V

• 8 Layer PCB
vs 6
• Rpacks
vs single Resistors

Cisco Systems Confidential FRS B2

4
Buddha (B2) vs Tahiti Differences

• 16MB EDO DRAM

vs 8MB
• Board ID 0x17
vs 0x16
• Eel flex system LEDs
vs Pelican flex LEDs & switch
• AVR controlled port LEDs
vs Muscle controlled

Cisco Systems Confidential FRS B2

5
Buddha (B2) vs Tahiti Differences

• AVR controlled Fan

vs fan control chip
• 5V on-board switcher
vs 3.3V
• gbics provided on a BRICK
vs on main PCB
• Hiearchical Schematics
vs flat schematics

Cisco Systems Confidential FRS B2

6
 SWITCH SYSTEM
CPU SYSTEM & CLOCKS FWD
RS232
GBIC BRICK RAM

uP PCI BUS Frank 0 GBIC

CPU
BUFFER Data RCVR

BUFFERS
RAM
DRAM
Buffer
FWD
RAM
Santa 2 CHANNELS
I/O Buffer
Cruz Avalon
Frank 1 GBIC
64
LED
RCVR
AVR NOTIFY
1 pair = channel RAM
FLASH RAM 14 channels

NOTIFY Manhattan ADDRESS

48 channels NOTIFY Manhattan ADDRESS
RAM TABLE RAM TABLE
0 ooo 5

Schematic: Bridge, 2x Quad,

8x Single.

Ports 41-48
Ports 1-8

Big Buddha
Cisco Systems Confidential FRS B2
7
Current Testing Methods

• Engineering
boot -post (Internal loop-back)
Perf 5 (Internal & External loop-back)
Debug Monitor Diagnostics
EMI 100 - Internal & external loop-back

• Manufacturing
Boot -post
Perf 5
BST (SmartBits)

Cisco Systems Confidential FRS B2

8
 CPU Subsystem

• U72 = Processor
• U76, U77 = EDO DRAM
• U68 = Flash Memory
• U106 = I/O Buffer
• U105, U39 = System LED Buffers
• U85 = RS232 Driver
• U75 = Muscle ASIC
• U69, U70, U73, U74 = Santa Cruz PCI
Buffers

Cisco Systems Confidential FRS B2

9
 CPU Subsystem functions

• Loads and executes the software code

from Flash and then DRAM
• Configures the ASICs
• Controls the LEDs
• During operation, supports user
intervention and dynamic
configurability

Cisco Systems Confidential FRS B2

10
 CPU Troubleshooting

• Power-on Failure: Symptom = No LEDs on at all.

Check 5V -- back left near C922, TP VDD.
Check 3.3V -- back right near U43, TP 3.3V.
• No System Clock: Symptom = Some LEDs ON but
do not change state.
Check 25MHz the clock at U61 pin 14. No signal = Santa
Cruz ASIC problem or 50MHz clock (U53) to Santa Cruz
problem. or missing R76 (25MHz source).

Cisco Systems Confidential FRS B2

11
 CPU troubleshooting cont ..
• Flash not programmed/corruption: Symptom =
LEDs ON but do not change state.
Probe U106 pin 19 @ POR, if only 4 low pulses are seen then
possible flash device problem. Check pins 11-18 on the same
buffer device during power-up to ensure that the signal swings
between <0.8V and >2.4V.

• DRAM Failure: Symptom = All port LEDs go

green, Port 1 LED turns OFF but Port 2 never
turns OFF.
The Port 2 LED will turn off if the DRAM is OK. Possible causes:
Device failure; solder problems; short on the Processor Data bus.

• Board ID - Defines which board the software is running on.

The value is defined by resistor settings. The value must be 17
Hex. Stuffed resistors: R693, R685, R680, R682, R686, R690, R675,
R677.

Cisco Systems Confidential FRS B2

12
 LED & Fan System

• U80, AVR, microcontroller

• U42, U82, U78, U79, glue logic
• Q7, diode drop for AVR supply
• R56, PCB temperature sensor
• U1-U3, U67, LED current source drivers
• U71, Fan NCH MOST driver

Cisco Systems Confidential FRS B2

13
 LED Trouble Shooting

• AVR, U80 not programmed or no CLK: Symptom

= LEDs random pattern but do not change state,
fan may be off, system LED blinking.
Probe U80 pin 8, if not 8MHz after power-up means poor
crystal circuit.
Probe U3 pin 3, if there are no pulses present the AVR
may not be programmed.

Symptom: Some port LEDs fail. Solution: Press on

problem LED or LED drivers (U1-U3, U67). If LED
lights there is a solder joint problem.

Cisco Systems Confidential FRS B2

14
 Fan Trouble Shooting

• AVR, U80 not programmed: see previous slide.

• Check fan connected and that fuse FS4 are okay.
• Check that AVR (U80) pin 13 is producing pulse
train. The low time varies with temperature.
• Check RA109, U71 and Fan.

Cisco Systems Confidential FRS B2

15
Loading Diagnostics - Backdoor

• Power-on the unit while pressing the Mode switch

• Wait until System LED blinks.
• Continue to hold switch until STAT system LED turn
on.
• Count two seconds, then release Mode switch
• At prompt type ‘flash_init’
• Type ‘dir flash:’ to see what’s in flash.
• Type ‘load flash:c3500xl-diag-mz-112.0.66-SA6’
• Type ‘?’ to see a list of commands.

Cisco Systems Confidential FRS B2

16
Santa Cruz Subsystem

• U56 = Santa Cruz III ASIC

• U55 = Notify RAM
• U45 = CPU Buffer
• U61 = 25MHz clock buffer

Cisco Systems Confidential FRS B2

17
 Santa Cruz functionality

• Connects the Processor to the switch

ASICs.
• Generates the 25MHz clock for the
Processor (U72), Muscle (U75) and the
AVR glue logic (U78).
• Santa Cruz is connected to the Notify
Ring.

Cisco Systems Confidential FRS B2

18
 Santa Cruz Troubleshooting

• Bad CPU Buffer - Execute ‘test_cpu_buffer’ to verify

• Bad CPU Notify RAM - Execute ‘test_cpu_notify’ to verify
• Bad switch communication channels
Execute ‘molecule_test’ to verify that Santa Cruz can talk to
Avalon.
A failure indicates a bad Avalon device or most probably
assembly problems. Check this out by probing the signals
from Santa Cruz to Avalon and visa versa. While probing
Santa Cruz output resistors (RA86), if no activity then Santa
Cruz assembly problem else while probing Avalon output
resistors (RA82), no activity means Avalon assembly
problem.

Cisco Systems Confidential FRS B2

19
 Avalon Subsystem

• U46 = Avalon ASIC

• U62-U65 = Data Buffer
• Buffer Table inside Avalon.
• U53, 50MHz and 100MHz clock source.

Cisco Systems Confidential FRS B2

20
 Avalon functionality

• Manages the “store and forward”

operation of the switch for all ports.
• Data Buffer - all frames are stored in
here.
• Avalon is NOT connected to the Notify
ring.

Cisco Systems Confidential FRS B2

21
 Avalon Troubleshooting

• Symptom: ‘molecule_test’ fails

If it is not a channel problem between Santa Cruz
and Avalon then verify the ASIC is operating by
probing U65 pin 27 (RAS#) to determine if a 20ns low
repeating pulse is present.
• Avalon Memory is tested using the monitor
‘TestAvalonBuffer’ command

Cisco Systems Confidential FRS B2

22
 Manhattan Subsystem
• Manhattan = Del Mar plus 8 PHYs
• U4-U9 Manhattan (Man.) ASIC, software uses
identifiers 0-5, ports 1-48.
• U10, U16, U30, U91, U97, U103 = Man. 0
Address Table SRAM
• U11, U17, U31, U90, U96, U102 = Man. 1
Address Table SRAM
• U12, U18, U32, U89, U95, U101 = Man 2 Address
Table SRAM
• U13, U19, U33, U88, U94, U100 = Man 3 Address
Table SRAM

Cisco Systems Confidential FRS B2

23
 Manhattan Subsystem Cont.

• U14,U20, U34, U87, U93, U99 = Man 4 Address

Table SRAM
• U15, U21, U35, U86, U92, U98 = Man 5 Address
Table SRAM
• U24-U29 = Man 0-5 Notify RAM
• U51 = 67MHz Oscillator
• U41, U38 = 50 MHz clock buffer

Cisco Systems Confidential FRS B2

24
 Manhattan Functionality

• Provides the Ethernet interface to the

switch memory.
• Address Table is used to “learn” new
Ethernet MAC addresses
• Is connected to the Notify ring.

Cisco Systems Confidential FRS B2

25
 Manhattan Troubleshooting
• Cannot access Address Table or Notify
RAM:
This is due to either a faulty ASIC or a bad channel 0
connection between the Manhattan and Avalon. Debug by
monitoring the channel signals at the source resistors.
» Man. 0 resistors are: RA68 (AV RC, ports 1 & 2); SC, RA3
» Man. 1 resistors are: RA60 (AV RC, ports 9 & 10); SC, RA9
» Man. 2 resistors are: RA39 (AV RC, ports 17 & 18); SC, RA15
» Man. 3 resistors are: RA41 (AV RC, ports 25 & 26); SC, RA21
» Man. 4 resistors are: RA43 (AV RC, ports 33 & 34); SC, RA27
» Man. 5 resistors are: RA45 (AV RC, ports 41 & 42); SC, RA33

Cisco Systems Confidential FRS B2

26
 Manhattan Troubleshooting

• Cannot access Address Table only:

check for a 67MHz clock problem on U41, U51
• Cannot access Notify RAM only: check
clock outputs from U41, U38
• All other channels (1-7) can only be debugged using
tests such as ‘emi100’, ‘emi10’ or ‘Perf5’

Cisco Systems Confidential FRS B2

27
 Ethernet Ports

• U4-U9 = PHY
• T1-T12 = Quad Transformers
• J1-2 = 8 over 8 -16 Gang RJ45
Connectors
• U51 = 67MHz Oscillator
• U41, U38 = 50 MHz clock buffer

Cisco Systems Confidential FRS B2

28
Ethernet Port Functionality

• Contains the analog and digital circuit to

make the cables talk in accordance with
the Ethernet electrical specifications
• Each port “talks” to switch through the
channel interface.

Cisco Systems Confidential FRS B2

29
 Ethernet Port Troubleshooting
• Check 50MHz clocks.
• Individual Port Failure:
This is the most common failure. The problem lies
with the PHY or the Transformer or the RJ45!
Sanity check: Use ‘emi10/100’ with the ‘internal loop-
back’ option to ensure that the PHY device
communicates correctly with the Manhattan(s).
- Are Rpacks missing from Manhattan or Avalon?

Cisco Systems Confidential FRS B2

30
 Ethernet Port Troubleshooting
cont ...
• Individual Port Failure cont…
Providing ‘internal’ feedback works then the problem lies with
the “front-end”
Front-end failures tracked down as follows, using loopback
cables:
1. Ex/ If Port 1 and Port 2 are connected and one port is failing
then look at pins 14 & 15 (port 1) and 18 & 16 (port 2) of the
Transformer to check that each port is transmitting.
2. Check pins 11 & 12 (port 1) and 20 & 19 (port 2) of the
Transformer to check that it is receiving.
3. If PHY is not transmitting or receiving then check power/gnd
connections made through ferrites reference designators = Zxx.
Most common parts to replace: PHY and Transformers!

Cisco Systems Confidential FRS B2

31
 Ethernet Port Troubleshooting
cont ...
• PERF Fails to start properly:
Occasionally, PERF fails to startup properly. This
can occur if PERF is started immediately after power-
up.

To fix this issue a “test perf stop” command.

Wait until all port LEDs are either GREEN (even port)
or AMBER (odd port).

Restart PERF “test perf start.”

Cisco Systems Confidential FRS B2
32
GBIC Subsystem
(BRICK)
U43 = Frank0 ASIC
U57 = Frank0 FWD SSRAM
U58 = Frank0 RCV SSRAM
U20 = Frank0 FWD SSRAM
U22 = Frank0 SERDES
U44 = Frank1 ASIC
U60 = Frank1 RCV SSRAM
Cisco Systems Confidential FRS B2
33
 GBIC Subsystem
(BRICK)

U59 = Frank1 FWD SSRAM

U23 = Frank1 SERDES
P1 = 40 pin connector to GBIC BRICK
U36, U37 = 125 MHZ CLOCK
OCSILLATORS
U66 = 50 AND 100 MHZ CLOCK
SOURCE

Cisco Systems Confidential FRS B2

34
Frank Functionality

• Provides the Ethernet Interface to the

Switch Memory.
• Is a Satellite like Manhattan.
• Frank is connected to the Notify Ring.

Cisco Systems Confidential FRS B2

35
Frank Functionality Continued

• External SSRAM
Forwarding RAM - contains Address Table,
Notification Tables, and Distribution FIFO
Receiving RAM - Receive FIFOs

• Requires 50, 100, and 125 MHz clocks.

Cisco Systems Confidential FRS B2

36
Front Ends

• Serdes 1.25Gb transceiver

• Transceiver communicates with Frank
• Transceiver takes 10-bit parallel data
from Frank and transmits 1-bit data
through the front end at 10x the parallel
buses rate

Cisco Systems Confidential FRS B2

37
 How things SHOULD work

• POST - checks Forwarding RAM and

Receive RAM, and performs a front
end in loop-back test.
• Frank is configured and RAMs are
initialized.

Cisco Systems Confidential FRS B2

38
When Things Do NOT Work -
Troubleshooting

• Module NOT recognized by switch

5V supply
3.3V supply
50MHz, 100MHz, clocks running U66, U41
125MHz clock U36, U37

Cisco Systems Confidential FRS B2

39
Troubleshooting Continued

• POST fails Forwarding RAM test

Correct orientation of RAM
3.3V present at U57, U59.
100 MHz appear at RAM U57, U59 pin 89.
RAM pins shorted or unconnected pins
Missing or wrong components near RAM
Bad RAM chip
Bad Frank PBGA ball connections to PCB
Bad Frank

Cisco Systems Confidential FRS B2

40
Troubleshooting Continued

• POST fails Receive RAM test

Correct orientation of RAMs
3.3V present at U58, U60.
100 MHz appear at RAMs U58, U60 pin89.
RAM pins shorted or unconnected pins
Missing or wrong components near RAMs
Bad RAM chip(s) - which address range(s) fail?
Bad Frank PBGA ball connections to PCB
Bad Frank
Cisco Systems Confidential FRS B2
41
Troubleshooting Continued

• POST fails Internal Loop-back test

Correct orientation of SERDES
3.3V present at SERDES.
125 MHz appears at SERDES U2, U3 pin 22.
SERDES pins shorted or unconnected pins
Missing or wrong components near SERDES
TTL signaling observed across parallel interface with Frank?
PECL signaling observed on the SERDES serial data lines?
Bad SERDES chip
Bad Frank PBGA ball connections to PCB
Bad Frank

Cisco Systems Confidential FRS B2

42
Troubleshooting Continued

• POST fails External Loop-back test

GBIC connector pins shorted or unconnected
5V appears at GBIC brick 20-pin connectors pin 16.
Poor contact on connector P1.
Missing or wrong components near GBIC
PECL signaling observed across serial interface
with GBIC?
Bad GBIC, GBIC connector or GBIC guide

Cisco Systems Confidential FRS B2

43