Troubleshooting ACI Fabric Discovery

This article describes steps to review and resolve issues when building an ACI fabric.
Assumptions:
** APIC IP information is defined in the APIC via KVM console
Notation:
** Bold blue information is entered by user
** Bold red information is to high-light content or errors.

1- APIC is not accessible as 'admin' user via KVM console.

• Login as ‘rescue-user’ and same password of ‘admin’ user
• Run ‘ip link’ to verify if bond0 (#6) and bond1 (#7) are created successfully and “UP”

admin@apic1:~> ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth1-1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond1 state
UP qlen 1000
link/ether 24:e9:b3:91:e9:58 brd ff:ff:ff:ff:ff:ff
3: eth1-2: <NO-CARRIER,BROADCAST,MULTICAST,SLAVE,UP> mtu 1500 qdisc mq master bond1
state DOWN qlen 1000
link/ether 24:e9:b3:91:e9:58 brd ff:ff:ff:ff:ff:ff
4: eth2-1: <BROADCAST,MULTICAST,SLAVE,UP> mtu 1500 qdisc mq master bond0 state UNKNOWN
qlen 1000
link/ether b8:38:61:f7:05:b1 brd ff:ff:ff:ff:ff:ff
5: eth2-2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state
UP qlen 1000
link/ether b8:38:61:f7:05:b1 brd ff:ff:ff:ff:ff:ff
6: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
link/ether b8:38:61:f7:05:b1 brd ff:ff:ff:ff:ff:ff
7: bond1: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master
oobmgmt state UP
link/ether 24:e9:b3:91:e9:58 brd ff:ff:ff:ff:ff:ff
8: oobmgmt: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
link/ether 24:e9:b3:91:e9:58 brd ff:ff:ff:ff:ff:ff
9: bond0.4093@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1496 qdisc noqueue state
UP link/ether b8:38:61:f7:05:b1 brd ff:ff:ff:ff:ff:ff

• Log onto the APIC CIMC and review the VIC adapter information
• Review VIC adapter firmware and the adapter link status
2- APIC is not accessible via Chrome/Firefox browser.
• ·Check the APIC product ID is “APIC-SERVER-L1” or similar and not UCSC-C220-M3S

3 - APIC unable to join cluster

• · Follow instructions to Update the APIC product ID to “APIC-SERVER”
• · An error as below might be shown for incorrect PID
 •

4- Switches are not discovered by APIC.

ACI fabric discovery starts at APIC controller when the first leaf is added followed by the spines, so it is
essential that the first leaf is discovered . If failed,

• Verify correct cabling

• As ‘admin’ user, verify the status of APIC with ‘acidiag fnvread’. If succeed, switch state will
show ‘active’; If there is issue, switch state shows ‘inactive’ or ‘discovering’
admin@apic1:~> acidiag fnvread

ID Name Serial Number IP

Address Role State LastUpdMsgId

------------------------------------------------------------------------
-------------------------

101 Leaf-
1 SAL1815Q3J0 10.0.36.92/32 leaf active 0

102 Leaf-
2 SAL17299NAD 10.0.36.95/32 leaf active 0

110 Spine-
1 SAL1811NN5K 10.0.36.94/32 spine active 0

111 Spine-
2 SAL1811NN64 10.0.36.93/32 spine active 0

admin@apic1:~> acidiag fnvread

ID Name Serial Number IP Address Role State LastUpdMsgId

-------------------------------------------------------------------------------------------------

102 Leaf-1 SAL1815PZ8G 10.0.8.95/32 leaf active 0

103 Spine1 SAL1824UGH8 10.0.8.94/32 spine inactive 0x100000002f

104 Spine-2 SAL1824UNPK 10.0.8.93/32 spine active 0

105 Leaf-2 SAL1815Q3B8 10.0.8.92/32 leaf active 0

Total 4 nodes
——————————————
admin@apic1:~> acidiag fnvread

ID Name Serial Number IP Address Role State LastUpdMsgId

-------------------------------------------------------------------------------------------------

• 101 Leaf-1 SAL1815Q3J0 0.0.0.0 leaf discovering 0

• verify with “show lldp neighbors” in APIC – remember using ‘esc’ twice following the
command. If switch is not shown in “lldp neighbors” output, check the VIC firmware and LLDP
status as well as cable connection
• Check VIC Adapter Properties from KVM Console and esure that LLDP is not enabled on the VIC.
This will blackhole LLDP traffic preventing discovery
• Make sure the infra VLAN matches by comparing 'show lldp toolkit in [eth2-1|eth1-1]' and 'show
lldp toolkit out [eth2-1|eth1-1]' from the APIC. Or from the Leaf check 'cat
/mit/sys/lldp/inst/summary'
admin@apic1:~> show lldp neighbors
Leaf-1 Specify Fabric Node Name
Leaf-2 Specify Fabric Node Name
Spine-1 Specify Fabric Node Name
Spine-2 Specify Fabric Node Name
node Fabric node

• As ‘admin’ user, verify with “show lldp neighbors”

Leaf-2# show lldp neighbors

Capability codes:

(R) Router, (B) Bridge, (T) Telephone, (C) DOCSIS Cable Device

(W) WLAN Access Point, (P) Repeater, (S) Station, (O) Other

Device ID Local Intf Hold-time Capability Port ID

apic1 Eth1/1 120 b8:38:61:f7:05:b1

Spine-1 Eth1/97 120 BR Eth1/1

Spine-2 Eth1/98 120 BR Eth1/1

Total entries displayed: 3

5- If switch is “inactive” - it means the switch was discovered but cannot be added to the ACI fabric,
verify SSL information.

The APIC won’t allow addition of any switches to the fabric if the SSL cert is not yet effective or the switch
does not have SSL cert.

• Check the status of the APIC SSL cert

admin@apic1:~> acidiag verifyapic


openssl_check: passed


file not found: /securedata/sshd_host_keys/ssh_host_rsa_key
installation_check.sh ERROR: ssh_check:

dsa/rsa key check failed
 • Check the status of SSL cert on the switch, log onto the switch as ‘root’ user and run

For TORs run: act_util keypair_show 0

For Spines run: act_util keypair_show 1

If installed, the command displays RSA and CERT key in binary codes.

• Check the system date of the APIC server as well as of the switches

admin@apic1:~> date

Fri Jul 11 13:04:17 UTC 2014

• As ‘root’ user, verify APIC SSL information:

root@apic1:~# openssl verify -CAfile /securedata/cacerts/cacert.crt /securedata/ssl/server.crt

/securedata/ssl/server.crt: OK

• If the SSL cert is not yet effective

openssl verify -CAfile /securedata/cacerts/cacert.crt /securedata/ssl/server.crt

WARNING: can't open config file: /usr/lib/ssl/openssl.cnf

/securedata/ssl/server.crt: serialNumber = PID:N9K-C9396PX SN:SAL1815Q3J0, CN = SAL1815Q3J0

error 9 at 0 depth lookup:certificate is not yet valid

• Verify the effective dates of the SSL Cert:

root@apic1:~# openssl x509 -noout -issuer -subject -dates -in /securedata/ssl/server.crt

issuer= /O=Cisco Systems/CN=Cisco Manufacturing CA

subject= /serialNumber=PID:APIC-SERVER-L1 SN:FCH1747V0SG/CN=FCH1747V0SG

notBefore=Jun 28 00:40:06 2014 GMT

notAfter=Jun 28 00:50:06 2024 GMT

 • SSL cert needs to be issued by “Cisco Manufacturing CA” as shown above. Run this command
to see more details of the cert: openssl asn1parse < /securedata/ssl/server.crt

• If SSL cert shows “Insieme Network”, the switch wont be added to the fabric. Follow the
instructions to install
SSL cert on the switch

6- Leaf is discovered, but not the spines, Run through steps 4 again, the same ‘SSL’ commands can be
used to verify the SSL cert status of the switches. The SSL cert needs to be effective per the switch system
date as well as the APIC system date. The SSL cert effective dates cannot be changed only the switch or
APIC system date can be changed. It is important that the system dates of these components are close
together and reflect the current date/time.

Example of a command to change the system date

date ‘071112302014’
to change the date to July 11 12:30 PM 2014.

6- The switch status is ‘discovering’ in ‘acidiag’ output, check the switch firmware, ensuring it has the
correct version for the APIC as well as matching on other switches.

7- If the switch had been part of another fabric/APIC, it wont be added to the new APIC until all
configuration is cleaned up.

• Verify the switch ‘summary’ if it has been part of another fabric. As ‘admin’ user, log onto the
switch.

cat /mit/sys/summary
# System
address : 10.0.36.95
childAction :
currentTime : 2014-07-11T14:49:54.936+00:00
dn : sys
fabricId : 1
fabricMAC : 00:22:BD:F8:19:FF
id : 102
inbMgmtAddr : 0.0.0.0
lcOwn : local
modTs : 2014-07-11T18:58:02.123+00:00
mode : unspecified
monPolDn : uni/fabric/monfab-default
name : Leaf-2
oobMgmtAddr : 0.0.0.0
podId : 1
rn : sys
role : leaf
serial : SAL17299NAD
state : in-service
status :
 systemUpTime : 00:21:50:31.000

• Follow instructions to clean up switch configuration and reload before adding it to the new
fabric
8- Verify the system logs for other errors

• cd /var/sysmgr/tmp_logs/
• tail –f svc_ifc_policyelem.log