SER2965BU

Advanced Troubleshooting of i bu t i o n
is t r
ESXi Server 6.x for vSphere o r d
t i o n
Gurus bl i c a
r p u
t fo
: N o
ten t
C o n
2 017
w o rld
V M

Ragavendra P Kumar & Abhilash Kunhappan

#VMworld #SER2965BU
Learner Objectives
• Improvise troubleshooting steps for a vSphere admin by

• Understanding the ESXi Architecture from a component breakdown perspective

t i o n
i s tr ibu
• Learn “7” important and key :- or d
a t i on
1. Log Files : which log files to review and when u blic
o r p
o t f
2. ESXi Commands t: N issues
: to isolate and troubleshoot
n
o nte
3. Configuration Files : forms17theCbackbone of troubleshooting sessions
d 2 0
w o rl
V M
Every VMworld session is time bound and we will cover as much as we can.
For all remaining questions, we are always a phone call away.

#SER2965BU CONFIDENTIAL 6
Agenda
© 

t i o n
1 7 Important Log files
i s tribu
or d
2 t ion
7 Key ESXi Commands
bli c a
r p u
3 7 Significant Configuration Files Not fo
en t:
C ont
1 7
o rl d 20
VM w

#SER2965BU CONFIDENTIAL 7
ESXi – Architecture
HTML 5 - vSphere
WebClient vpxd DRS
VIM Clients VIM FT
HA Guest OS & Apps
HTML 5 – ESXi Power
CLI
t i o n
ibu
Host Client vCPU
vCenter Server
di s tr
n or
io
vpxa

c a t
bli
Hostd Sys
FDM
Calls
p u
User world Support
r
VMX
o t fo
esxcli Process
n t N
: Scheduler
localcli

ont e

VMK Call
Shell User worlds Resource

1 7 C VMM

d 20 Monitor
l
vNIC

or Network Storage

VMw
Stack Stack vSCSI
vSCSI
DVFilter

vmkapi and Interrupt handler

VMkernel

Server Hardware

8
#SER2965BU CONFIDENTIAL
ESXi Log Files


t i o n
i s tribu
or d
t ion
bli c a
r p u
o t fo
n t: N
ont e
1 7 C
o rl d 20
VM w

#SER2965BU CONFIDENTIAL 9
1. vmksummary.log – Abrupt Host reboots
• Time stamp indicating ESXi server reboot

t i o n
i s tribu
or d
t ion
bli c a
r p u
fo
n Not shows host rebooting has not been tracked
• VMkernel will always track Host reboots, the below
t:
ont e
1 7 C
o rl d 20
VM w

• Disable the Automated Server Recovery feature from the BIOS to avoid automatic reboots

#SER2965BU CONFIDENTIAL 10
2. boot.gz log - Slow Boot Issues
• To troubleshoot both slow boot and boot process hang of ESXi server

• Location : /var/log/boot.gz n
t i o
ibu i s t r
• Use DCUI - Alt + F12 to review VMkernel messages during this time o r d
t io n
li c a
bscreen
•
r u
Enabling Serial logging using (Shift + o) at the start pup for log redirection
o t fo
– vmkBootVerbose=TRUE vmbLog=TRUEnt : N
logPort=com1
o nte
1 7 C no-auto-partition bootUUID=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
d 2 0
– debugLogToSerial=1 logPort=com1

w o rl
VM

#SER2965BU CONFIDENTIAL 11
2. boot.gz log - Slow Boot Issues
• Slow ESXi Boot Issues when, RDM devices are not perennially tagged
• Validate the esx.conf file with a key word search “perennialResvd” with respect to the NAA.ID of
the LUN t i o n
i s tribu
or d
t ion
bli c a
r p u
o t fo
n t: N
ont e
1 7 C
o rld 20
• Command to set w
VMthe perennially reserve tag per LUN
# esxcli storage core device setconfig -d naa.id --perennially-reserved=true

#SER2965BU CONFIDENTIAL 12
3. hostd & hostd-probe.log – ESXi Not Responding
• hostd unresponsive and Host Not responding from vCenter server
• A successful hostd process start up or working fine hostd process should finish reading and
populating the “vmInventory.xml” file t i o n
i s tribu
or d
t ion
bli c a
r p u
o t fo
n t: N
ont e
1 7 C
o rl d 20
VM w
• Direct Connect to the ESXi with Host client should also validate the hostd status
• Do NOT restart the ESXi Server when hostd is in a loop if you need to pursue the cause

#SER2965BU CONFIDENTIAL 13
3. hostd.log – ESXi Not Responding
• Understanding the Hostd status and validating the Hostd service restart is successful

t i o n
i s tribu
or d
t ion
bli c a
r p u
t fo
No
• Key words search in hostd.log – “BEGINt:SERVICES”
n
nt e
17 Co
2 0
w o rld
V M

#SER2965BU CONFIDENTIAL 14
4. vmware.log - Virtual Machine Issues
• VM Power ON issues
• Attempt a Virtual Machine power ON and then validate the below log
t i o n
• Identify the cause of the Power ON failure
i s tribu
or d
t ion
bli c a
r p u
o t fo
n t: N
ont e
1 7 C
o rl d 20
VM w

#SER2965BU CONFIDENTIAL 15
5. vmkernel.log - Storage Issues
Understanding SCSI Sense codes are key for Storage issues

t i o n
i s tribu
or d
t ion
bli c a
r p u
o t fo
n t: N
ont e
1 7 C
o r l d 20
VM w
• SCSI codes which are T10 org defined codes
http://www.t10.org/lists/2op.htm
• SCSI sense codes
http://www.virten.net/vmware/esxi-scsi-sense-code-decoder/

#SER2965BU CONFIDENTIAL 16
6. vobd.log - Network & Storage Issues
• Network showing uplink transition

t i o n
i s tribu
or d
t ion
bli c a
r p u
o t fo
n t: N
• Storage disconnect messages
ont e
1 7 C
o rl d 20
VM w

#SER2965BU CONFIDENTIAL 17
7. fdm.log – High Availability (HA) Issues

to identify :-
• the HostID?
t i o n
• the Host is a Slave or Master?
i s tribu
• the Host that was isolated or d
t ion
from cluster?
bli c a
r p u
o t fo
n t: N
ont e
1 7 C
o rl d 20
VM w

#SER2965BU CONFIDENTIAL 18
ESXi Commands
©

t i o n
i s tribu
or d
t ion
bli c a
r p u
o t fo
n t: N
ont e
1 7 C
o rl d 20
VM w

#SER2965BU CONFIDENTIAL 19
1. To Monitor & Configure ESXi Server Settings…
boot vib
module 03 04 profile
process sources t i o n
i s tribu
or d
nic t ion
bli c a
firewall 02
r p u cluster
vSwitch t f o 05 network
: N o
en t datastore
C ont
1 7
o rl d 20
san VM w list
vvol 01 06 kill
vmfs process
esxcli

#SER2965BU CONFIDENTIAL 20
# esxcli Displaying the ARP
and Neighbor
Discovery cache

t i o n
i s tribu
or d
t ion
bli c a
r p u
VMX World
o t fo
ID
n t: N
information VMX Cartel
o nt e
1 C
7 ID
0
d 2 information
o rl
VM w

#SER2965BU CONFIDENTIAL 21
2. VMkernel Sysinfo Shell Command…
** # vsish
get <opt.1>/<opt.2>/<opt.3>,….
t i o n
i s tribu
or d
t ion
bli c a
hardware netfor pu storage
n t: Not
ont e
1 7 C
o rl d 20
VM w
/bios /pNics/vmnic<>/stats /scsifw/devices/.<>/stats
/hardwareInfo /portsets/vSwitch<>/stats /scsifw/adapters/.<>/stats

** take assistance always from GSS when in doubt #SER2965BU CONFIDENTIAL 22

# vsish

t i o n
i s tribu
or d
t ion
bli c a
r p u
o t fo Get the VMX
n t: N
ont e Cartel ID

1 7 C
o rl d 20
VM w
Get the
GroupID

#SER2965BU CONFIDENTIAL 23
 3. To Manage ESXi & VM Configurations…
login auth
connect 03 04 task_list
registervm task_cancel
t i o n
i s tribu
or d
net t ion
bli c a port_info
r pu
storage 02
firmware
o t fo 05 service_list

n t: N remove_service
ont e
1 7 C
o r l d 20
getallvms VM w vprobes
power.getstate 01 06 shutdown
get.guest refresh_pnic
vim-cmd

#SER2965BU CONFIDENTIAL 24
# vim-cmd
Validates if the Hostd
is running by listing
the VM’s registered

t i o n
i s tribu
or d
a t i on
p ublic List the tasks running
fo r on the ESXi host
n t: Not
ont e
1 7 C
o rl d 20
VM w

#SER2965BU CONFIDENTIAL 25
4. To Manage VMFS Volumes & Virtual Disks…

# vmkfstools <opt.1> name.vmdk <opt.2>

t i o n
i s tr ibu
or d
t ion
bli c a -x
r pu
-e -t10 .

-v10 o t fo check,
n t: N-v10 repair
ont e
1 7 C
o rl d 20
VM w
to check VM to check Inode to check & correct
Snapshot Chain integrity of a vmdk basic file level
consistency like file in VMFS layer issues of a vmdk
PID & CID in and not in GOS file in VMFS layer
vmdks level

#SER2965BU CONFIDENTIAL 26
# vmkfstools VM Snapshot Chain
Consistency check

t i o n
i s tribu
or d
t ion
bli c a
r p u
o t fo
n t: N
ont e
1 7 C Check or Repair

o rl d 20 the Virtual Disk

VM w

#SER2965BU CONFIDENTIAL 27
5. # memstats – to Get Detailed Memory Statistics
• VM is identified by the VMX cartel ID using the command ps | grep –i <vmx name>
• Memstats –r vm-stats –s name:<choose the options needed>

Get the VMX cartel

ID t i o n
i s tribu
Virtual Machine Stats check o r d
t n
io ID
based on the VMX
bli c acartel
p u
t f or
: N o
t en t
C on
1 7
o rl d 20
VM w

#SER2965BU CONFIDENTIAL 28
# memstats
• VM is identified by the VMX cartel ID using the command ps | grep –i <vmx name>
• Memstats –r swap-stats –s name:<choose the options needed> -u mb
Swap Statistics on the
Virtual Machine
t i o n
i s tribu
or d
t ion
bli c a
r p u
o t fo
n t: N
ont e
1 7 C
o rl d 20
VM w

#SER2965BU CONFIDENTIAL 29
6. # pktcap-uw - Network Packet Capture Analysis Tool
• pktcap-uw --trace -- vmk <vmk port> Output can be redirected to
• pktcap-uw --trace -- uplink <uplink name> a file as well

t i o n
i s ibu
tr path from
o d
Shows packet traverse
r to identify drop
io
each layer
t n and
c a
bli within the ESXi host
r p u
o t fo
n t: N
ont e MAC address of the Port
1 7 C that is being captured

o rl d 20
VM w

#SER2965BU CONFIDENTIAL 30
7. To Monitor Core ESXi Performance…
virtual machine
v
Power
(usage & management) CPU t i o n
p c
i s tr ibu
or d
t ion
bli c a
r p u
o fo
t
network n n t N
:esxtop m
ont e memory
(vSwitch, uplinks)
1 7 C
o r l d 20
VM w

x d
vSAN
disk
u (hba level info)
storage
(LUN specific) #SER2965BU CONFIDENTIAL 31
# esxtop

t i o n
i s tribu
or d
t ion
bli c a
r p u
o t fo
n t: N
ont e
1 7 C
o rl d 20
VM w

#SER2965BU CONFIDENTIAL 32
ESXi – 7 Effective Commands for Troubleshooting
1. esxcli : Monitoring or configuring ESXi infra settings

2. vim-cmd
on
: Managing ESXi server & Virtual Machine related configurations
u ti
s tr i b
di
3. vsish : VMkernel Sys Info Shell
n or
c a t io
p bli
uvirtual disks from command line
4. vmkfstools : for management of VMFS volumes
o r &
f
N o t
n t:
5. memstats : Memory statisticsonte
1 7 C
d 2 0
6. pktcap-uw o rl
: Anwenhanced Network packet capture analysis tool
VM
7. esxtop : Real time performance monitoring

#SER2965BU CONFIDENTIAL 33
ESXi Configuration Files


t i o n
i s tribu
or d
t ion
bli c a
r p u
o t fo
n t: N
ont e
1 7 C
o rl d 20
VM w

#SER2965BU CONFIDENTIAL 34
7 Configuration File
1. /etc/vmware/esx.conf : Holds information about Storage, Network and Hardware

2. /etc/vmware/hostd/vmInventory.xml : Virtual Machine Inventory list update on the ESXi

t i o n
ribu
3. /etc/vmware/hostd/authorization.xml dis–tauthorization
: vCenter to ESXi host connection
o r
o n
4. /etc/vmware/vpxa/vpxa.cfg : vCenter and ESXi cati
ubli connectivity
o r p
N tf
o configuration file
5. /etc/vmware/vmkiscsid/iscsid.conf
nt
: :
iSCSI
o nte
17 C
6. /etc/vmware/fdm 2 0 : FDM config file with Clusterconfig, Hostlist and VM metadata
w orld
7. V M
/etc/vmware/license.cfg : License configuration file for ESXi host

#SER2965BU CONFIDENTIAL 35