Professional Documents
Culture Documents
Vsphere4 Storage
Vsphere4 Storage
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
Agenda
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
About the Speaker
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
Collecting Logs – User Interface
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
vSphere 4.0 Event Handling
vSphere provides a framework that defines the following use cases
Each of the use cases link to corresponding KBs that describe where the error occurred
(e.g. affected vmnic#, portgroup, vSwitch, storage path etc.) and provides
troubleshooting tips to fix the issue
Updates to these events will be provided as part of vSphere 4.0 updates
Events also logged to /var/log/vmkernel (or /var/log/messages on ESXi) with VOB prefix
Networking
vprob.net.connectivity.lost
vprob.net.redundancy.lost
vprob.net.redundancy.degraded
vprob.net.e1000.tso6.notsupported
Storage
vprob.storage.connectivity.lost
vprob.storage.redundancy.lost
vprob.storage.redundancy.degraded
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
Events Use Cases
VMFS specific:
vprob.vmfs.nfs.server.disconnect
vprob.vmfs.nfs.server.restored
vprob.vmfs.heartbeat.timedout
vprob.vmfs.heartbeat.recovered
vprob.vmfs.heartbeat.unrecoverable
vprob.vmfs.lock.corruptiondisk
vprob.vmfs.resource.corruptiondisk
vprob.vmfs.volume.locked
Migration Specific:
vprob.net.migrate.vmknic
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
Networking vProb
vprob.net.connectivity.lost
http://kb.vmware.com/kb/1009515
Connectivity to a physical network has been lost, all the affected portgroups
are part of the message (e.g. >Lost network connectivity on virtual switch
"system". Physical NIC vmnic1 is down. Affected port groups: "COS", "VM
Network".<)
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
Networking vProb
vprob.net.redundancy.lost
http://kb.vmware.com/kb/1009542
Only one physical NIC is currently connected, one more failure will result in a
loss of connectivity (e.g. >Lost uplink redundancy on virtual switch "system".
Physical NIC vmnic0 is down. Affected port groups: "COS", "VM Network".<)
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
Networking vProb
vprob.net.redundancy.degraded
http://kb.vmware.com/kb/1009546
One of the physical NICs in your NIC team has gone down, you still
have n-1 NICs available (e.g. Uplink redundancy degraded on virtual
switch "vSwitch0". Physical NIC vmnic1 is down. 2 uplinks still up.
Affected portgroups: "VM Network“.)
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
Networking vProb
vprob.net.e1000.tso6.notsupported
http://kb.vmware.com/kb/1009546
Guest e1000 driver is misbehaving and sending TSO IPv6
packets, which will be dropped. The vProb specifies the affected
VM, and the KB article discusses ways to fix this.
"Guest-initiated IPv6 TCP Segmentation Offload (TSO) packets
ignored. Manually disable TSO inside the guest operating system in
virtual machine "XYZ", or use a different virtual adapter."
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
Storage vProb
vprob.storage.connectivity.lost
http://kb.vmware.com/kb/1009553
The connectivity to a specific device has been lost (e.g. "Lost connectivity to
storage device naa.60a9800043346534645a433967325334. Path
vmhba35:C1:T0:L7 is down")
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
Storage vProb
vprob.storage.redundancy.lost
http://kb.vmware.com/kb/1009554
Only one path is remaining to a device and you no longer have any
redundancy (e.g. "Lost path redundancy to storage device
naa.60a9800043346534645a433967325334. Path vmhba35:C1:T0:L7 is
down.")
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
Storage vProb
vprob.storage.redundancy.degraded
http://http://kb.vmware.com/kb/1009555
One of your paths to a device has been lost but you still have n-1
paths remaining (e.g. "Path redundancy to storage device
naa.60a9800043346534645a433967325334 degraded. Path
vmhba35:C1:T0:L7 is down. 3 remaining active paths.")
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
VMFS vProb
vprob.vmfs.nfs.server.disconnect
http://kb.vmware.com/kb/1009566
vprob.vmfs.nfs.server.restored
http://kb.vmware.com/kb/1009568
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
VMFS vProb
vprob.vmfs.heartbeat.timedout
http://kb.vmware.com/kb/1009557
VMFS Volume Connectivity Degraded 496befed-1c79c817-6beb-
001ec9b60619 san-lun-100
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
VMFS vProb
vprob.vmfs.heartbeat.recovered
http://kb.vmware.com/kb/1009556
VMFS Volume Connectivity Restored 496befed-1c79c817-6beb-
001ec9b60619 (san-lun-100)
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
VMFS vProb
vprob.vmfs.heartbeat.unrecoverable
http://kb.vmware.com/kb/1019559
VMFS Volume Connectivity lost 496befed-1c79c817-6beb-
001ec9b60619 san-lun-100
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
VMFS vProb
vrpob.vmfs.lock.corruptiondisk
http://kb.vmware.com/kb/1019565
Volume 4976b16c-bd394790-6fd8-00215aaf0626 (san-lun-100) may
be damaged on disk. Corrupt lock detected at offset 0
vprob.vmfs.resource.corruptiondisk
http://kb.vmware.com/kb/1019569
Volume 4976b16c-bd394790-6fd8-00215aaf0626 (san-lun-100) may
be damaged on disk. Resource cluster metadata corruption detected
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
VMFS vProb
vprob.vmfs.volume.locked
http://kb.vmware.com/kb/1019570
Volume on device naa.60060160b3c018009bd1e02f725fdd11:1
locked, possibly because remote host 10.17.211.73 encountered an
error during a volume operation and couldn’t recover.
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
Migration Specific
vprob.net.migrate.vmknic
http://kb.vmware.com/kb/1009550
The ESX advanced config option /Migrate/Vmknic is set to an invalid
Vmknic: vmk0. /Migrate/Vmknic specifies a Vmknic that VMotion
binds to for improved performance. Please update the config option
with a valid Vmknic or, if you don't want VMotion to bind to a specific
Vmknic, remove the invalid Vmknic and leave the option blank.
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
vProb KB Mapping
VProb ID KB #
vprob.net.connectivity.lost 1009515
vprob.net.redundancy.lost 1009542
vprob.net.redundancy.degraded 1009546
vprob.net.e1000.tso6.notsupported 1009548
vprob.net.migrate.bindtovmk 1009550
vprob.storage.connectivity.lost 1009553
vprob.storage.redundancy.lost 1009554
vprob.storage.redundancy.degraded 1009555
vprob.vmfs.heartbeat.recovered 1009556
vprob.vmfs.heartbeat.timedout 1009557
vprob.vmfs.heartbeat.unrecoverable 1009559
vprob.vmfs.lock.corruptiondisk 1009565
vprob.vmfs.nfs.server.disconnect 1009566
vprob.vmfs.nfs.server.restored 1009568
vprob.vmfs.resource.corruptiondisk 1009569
vprob.vmfs.volume.locked 1009570
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
VMware ESX 3.5/4.0 common Logs
(collected via vm-support)
Most logs are located under /var/log directory.
Locations listed here are relative to that directory
vmkernel vmware/esxcfg-boot.log
messages vmware/esxcfg-firewall.log
dmesg vmware/vmware-cim.log
boot.log vmware/esxupdate.log
initrdlogs/* oldconf/esx.conf.*
vmksummary rpmpkgs
vmware/hostd.log vmkernel-version
vmware/vpx/vpxa.log
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
VMware ESXi 3.5/4.0 Common Log Files
config.log vmware/hostd.log
messages vmware/aam/*
slpd.log vmware/vpx/vpxa.log
wsmand.log
configRP.log
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
VMware ESX 4.0 Specific Log Files
esxcfg-* and esxcli commands outputs
Stored in /tmp or a directory referenced in
/tmp/working_dir.txt of vm-support dump
Usually in /usr/lib/vmware/hostd/docroot/downloads
esxcfg-dumppart esxcfg-swiscsi
esxcfg-info esxcfg-vmknic
esxcfg-info-xml esxcfg-vswitch
esxcfg-mpath-devices esxcli-corestorage-claimrules
esxcfg-mpath-paths esxcli-nmp-devices
esxcfg-nas esxcli-nmp-paths
esxcfg-nics esxcli-nmp-satp-rules
esxcfg-resgrp esxupdate-patch-history
esxcfg-route esxupdate-vib-view
esxcfg-route6
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
VMware ESX 4.0 Specific Log Files
esx_install.log
•Located in /var/log directory
•sym-linked to /var/log/weasel.log
•All events from upgrade or fresh install
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
ESX 4.0 COS Location
COS VMDK
Example:
/vmfs/volumes/storage1/esxconsole-4a0a1e4f-f9fa-ef4e-70fa-001ec9b60619/esxconsole.vmdk
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
VMware ESX 4.0 Specific Log Files
boot-log directory
• Located in /var/log directory
• Replaced /var/log/initrdlogs (ESX 3.x)
• sym-linked to /vmfs/volume/<COS-home>/esxconsole-
<UUID>/logs
• contains
• sysboot-dmesg-boot.log
• sysboot-dmesg-late.log
• sysboot.log
• sysboot-vmkernel-boot.log
• sysboot-vmkernel-late.log
Events generated from vmkernel boot image
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
VMware ESX 4.0 Specific Log Files
Sample sysboot.log
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
VMware ESX 4.0 Specific Log Files
Sample sysboot.log
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
VMware ESX 4.0 Specific Log Files
Sample sysboot.log
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
VMware ESX 4.0 Specific Log Files
Sample sysboot.log
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
vmkernel Log
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
VMware ESXi 4.0 Specific Log Files
/var/log
• messages
• Equivalent to /var/log/vmkernel on ESX 4.0
• sysboot.log
• Same as ESX 4.0
• The rest of boot logs do not apply to ESXi 4.0
/var/log/vmware
• Same as VMware ESX 4.0
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
Messages Log Files (VMware ESXi 4.0)
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
SCSI Errors Verbose logging
VMware ESX and VMware ESXi 4.0 are enabled by default for
reporting the SCSI Error Strings “Scsi.LogCmdErros” (set to “1”)
To increase the verbosity of MultiPath events logging, ESX and
ESXi 4.0 advanced vmkernel option “Scsi.LogMPCmdErrors” is
enabled by default (set to “1”)
It is recommended that you redirect the logs to a vMA appliance
or a syslog host to collect historic logs and (for ESXi 4.0) to
preserve them between reboots
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
Redirecting VMware ESX/ESXi Logs to vMA
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
Redirecting VMware ESX/ESXi 4.0 Server Logs to vMA
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
Redirecting VMware ESXi 4.0 Logs
to a syslog server
Log on to VMware ESXi directly using vSphere Client
Configuration Advanced Options Syslog Remote
Enter the syslog server‟s host name
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
vmkernel Log - Components
Jun 19 09:12:54 giza vmkernel: 14:22:31:50.009 cpu3:1033)scsi-qla0: Scheduling SCAN for new luns....
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
Rescan Event - LUN Discovery – ESX 4.0
ScsiScan: 839: Path 'vmhba2:C0:T0:L21': Vendor: 'EMC ' Model: 'SYMMETRIX '
Rev: '5772'
ScsiScan: 842: Path 'vmhba2:C0:T0:L21': Type: 0x1f, ANSI rev: 4, TPGS: 0 (none)
ScsiPath: 3685: Plugin 'NMP' claimed path 'vmhba2:C0:T0:L21'
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
Rescan Event – LUN Discovery - ESX 4.0
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
Rescan Event – LUN Discovery - ESX 4.0
Device Type Codes
Code Description
00h Direct access block device
01h Sequential-access device (e.g. Tape Drive)
02h Printer device
03h Processor device
04h Write-once device
05h CD/DVD device
08h Tape Library
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
LUN Discovery - ESX 4.0 - More Examples
Path 'vmhba2:C0:T1:L0': Vendor: 'DGC ' Model: 'RAID 5 ' Rev: '0428„
Path 'vmhba2:C0:T1:L0': Type: 0x0, ANSI rev: 4, TPGS: 0 (none)
Path 'vmhba2:C0:T2:L0': Vendor: 'IBM' Model: 'ULT3580-TD4' Rev: '89B2„
Path 'vmhba2:C0:T2:L0': Type: 0x1, ANSI rev: 3, TPGS: 0 (none)
Path 'vmhba2:C0:T2:L1': Vendor: 'IBM' Model: '3573-TL ' Rev: '7.30„
Path 'vmhba2:C0:T2:L1': Type: 0x8, ANSI rev: 5, TPGS: 0 (none)
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
vmkernel log entries - Understanding SCSI Error Strings
Format:
Device/Host Sense_buffer[2] 12 13
Abbrev: D/H S ASC ASCQ
Extended:
“Device Status”/”Host Status” “Sense Key” “Additional Sense
Code” “Additional Sense Code Qualifier”
Example:
2/0 0x6 0x29 0x0
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
vmkernel log entries - Understanding SCSI Error Strings
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
Understanding SCSI Error Strings – Device Status
Device Status: (Displayed in decimal values)
Code Meaning
0 No errors
2 Check Condition
8 Device Busy
24 Reservation Conflict
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
Understanding SCSI Error Strings – Host Status
Host Status (displayed in decimal values)
Code Meaning
0 Host_OK
1 Host No_Connect
2 Host_Bus_Busy
3 Host_Timeout
4 Host_Bad_Target
5 Host_Abort
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
Understanding SCSI Error Strings –
Host Status (cont.)
Host Status (displayed in decimal or Hex values) listed here
in decimal
Code Meaning
6 Host_Parity
7 Host_Error
8 Host_Reset
9 Host_Bad_INTR
10 Host_PassThrough
11 Host_Soft_Error
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
Understanding SCSI Error Strings – Sense Key
Sense Key (displayed in hex)
Code Meaning
0x0 No Sense Information
0x1 Last command completed but used error correction
0x2 Unit Not Ready
0x3 Medium Error
0x4 Hardware error
0x5 ILLEGAL_REQUEST (Passive SP)
0x6 LUN Reset
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
Understanding SCSI Error Strings –
Sense Key – (cont.)
Sense Key (displayed in hex)
Code Meaning
0x7 Data_Protect – Access to data is blocked
0x8 Blank_Check – Reached an unexpected region
0xa Copy_Aborted
0xb Aborted_Command – Target aborted command
0xc Comparison for SEARCH DATA unsuccessful
0xd Volume_Overflow – Medium is full
0xe Source and Data on Medium do not agree
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
Understanding SCSI Error Strings – ASC/ASCQ
ASC and ASCQ are always in pairs (in hex) (ASQ usually 0)
Code Meaning
0x4 Unit Not Ready
0x3 Unit Not Ready – Manual Intervention Required
0x2 Unit Not Ready - Initializing Command Required
0x29 Device Power on or SCSI Reset
0x8b ASC_QUIESCENCE_HAS_BEEN_ACHIEVED (IBM FAStT)
0x94 ASC_Invalid_Req_due_To_Current_LU_Ownership (IBM FAStT)
0x01 ASCQ_Invalid_Req_due_To_Current_LU_Ownership (IBM FAStT)
0x02 ASCQ_QUIESCENCE_HAS_BEEN_ACHIEVED (IBM FAStT)
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
Understanding SCSI Error Strings - Examples (4.0)
Plugin Code Translation
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
Understanding SCSI Error Strings - Examples (4.0)
nmp_CompleteCommandForPath:
Command 0x2a (0x410004237c00) to NMP device
"naa.60060160432017005c97aea1b32fdc11" failed on physical
path "vmhba0:C0:T1:L1" H:0x1 D:0x0 P:0x0 Possible sense
data: 0x0 0x0 0x0.
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
Understanding SCSI Error Strings - Examples (4.0)
NMP: nmp_CompleteCommandForPath: Command 0x25 (0x410004134a80)
to NMP device "naa.60060160b4111600826120bae2e3dd11" failed on
physical path "vmhba0:C0:T1:L0" H:0x0 D:0x2 P:0x0 Valid sense
data: 0x6 0x29 0x0.
NMP: nmp_CompleteCommandForPath:
Command 0x25 to device "naa.60060160b4111600826120bae2e3dd11"
failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0.
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
Understanding SCSI Error Strings - Examples (4.0)
ScsiDeviceIO: 747: Command 0x16 to device
"naa.6006016004351f00bc913175af58de11" failed H:0x0 D:0x8 P:0x0
Possible sense data: 0x0 0x0 0x0.
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
Understanding SCSI Error Strings - Examples (4.0)
NMP: nmp_CompleteCommandForPath: Command 0x25 (0x4100051f7bc0) to
NMP device " naa.60060160e6341f00988aef0a4f4fde11" failed on
physical path "vmhba0:C0:T1:L2" H:0x0 D:0x2 P :0x0 Valid sense
data: 0x6 0x3f 0xe.
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
Understanding SCSI Error Strings - Examples (4.0)
NMP: nmp_DeviceAttemptFailover: Retry world failover device
"naa.600601600ade2200aa3f43e90e7dde11" - failed to issue command
due to Not found (APD), try again...
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
Identifying path state
fc.2001001b32351707:2101001b32351707-fc.50060160c1e04536:5006016841e04536-
naa.600601608cc01c002d45db8b0fb8dd11
Runtime Name: vmhba2:C0:T1:L3
Device: naa.600601608cc01c002d45db8b0fb8dd11
Device Display Name: DGC Fibre Channel Disk (naa.600601608cc01c002d45db8b0fb8dd11)
Group State: standby
Storage Array Type Path Config:
Path Selection Policy Path Config: {non-current path}
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
Identifying Path State - cont.
fc.2001001b32351707:2101001b32351707-fc.50060160c1e04536:5006016141e04536-
naa.600601608cc01c002d45db8b0fb8dd11
Runtime Name: vmhba2:C0:T0:L3
Device: naa.600601608cc01c002d45db8b0fb8dd11
Device Display Name: DGC Fibre Channel Disk (naa.600601608cc01c002d45db8b0fb8dd11)
Group State: active
Storage Array Type Path Config:
Path Selection Policy Path Config: {current path}
fc.2000001b32151707:2100001b32151707-fc.50060160c1e04536:5006016041e04536-
naa.600601608cc01c002d45db8b0fb8dd11
Runtime Name: vmhba1:C0:T0:L3
Device: naa.600601608cc01c002d45db8b0fb8dd11
Device Display Name: DGC Fibre Channel Disk (naa.600601608cc01c002d45db8b0fb8dd11)
Group State: active
Storage Array Type Path Config:
Path Selection Policy Path Config: {non-current path}
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
Any Questions?
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
Extra Slides
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
Appendix -ESX 4.0 Failover Logs With
Additional Logging
14:13:47:59.878 cpu7:4109)NMP: nmp_HasMoreWorkingPaths: STANDBY path(s) only to device
"naa.60060160432017005c97aea1b32fdc11".
14:13:47:59.887 cpu7:4374)WARNING: NMP: nmp_PspSelectPathForIO: Plugin VMW_PSP_MRU selectPath() returned
path "vmhba0:C0:T1:L1" for device "naa.60060160432017005c97aea1b32fdc11" which is in state standby
instead of ON. Status is Bad parameter
14:13:47:59.887 cpu7:4374)WARNING: NMP: nmp_SelectPathAndIssueCommand: PSP select path "vmhba0:C0:T1:L1"
in a bad state on device "naa.60060160432017005c97aea1b32fdc11".
14:13:47:59.887 cpu7:4374)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x410004237c00) to NMP device
"naa.60060160432017005c97aea1b32fdc11" failed on physical path "vmhba0:C0:T1:L1" H:0x1 D:0x0 P:0x0
Possible sense data: 0x0 0x0 0x0.
14:13:47:59.887 cpu7:4374)WARNING: NMP: nmp_DeviceRetryCommand: Device
"naa.60060160432017005c97aea1b32fdc11": awaiting fast path state update for failover with I/O
blocked...
14:13:47:59.887 cpu7:4374)WARNING: NMP: nmp_DeviceStartLoop: NMP Device
"naa.60060160432017005c97aea1b32fdc11" is blocked. Not starting I/O from device.
14:13:48:00.069 cpu7:4109)NMP: nmp_DeviceUpdatePathStates: Activated path "vmhba0:C0:T1:L1" for NMP
device "naa.60060160432017005c97aea1b32fdc11".
14:13:48:00.888 cpu1:4206)WARNING: NMP: nmp_DeviceAttemptFailover: Retry world failover device
"naa.60060160432017005c97aea1b32fdc11" - issuing command 0x410004237c00
14:13:48:00.888 cpu2:4373)WARNING: NMP: nmp_CompleteRetryForPath: Retry command 0x2a (0x410004237c00) to
NMP device "naa.60060160432017005c97aea1b32fdc11" failed on physical path "vmhba0:C0:T1:L1" H:0x0
D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0.
14:13:48:00.888 cpu2:4373)WARNING: NMP: nmp_CompleteRetryForPath: Retry world restored device
"naa.60060160432017005c97aea1b32fdc11" - no more commands to retry
14:13:48:00.888 cpu2:4373)ScsiDeviceIO: 746: Command 0x2a to device
"naa.60060160432017005c97aea1b32fdc11" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0.
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
Last slide magnified
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
Appendix - ESX 4.0 Failover Logs – FC
cable unplugged
14:13:32:16.716 cpu3:4099)<6>qla2xxx 003:00.1: LOOP DOWN detected mbx1=2h mbx2=5h mbx3=0h.
14:13:32:24.425 cpu6:4195)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x410004286980) to NMP
device "naa.60060160432017005c97aea1b32fdc11" failed on physical path "vmhba1:C0:T0:L1" H:0x5 D:0x0
P:0x0 Possible sense data: 0x2 0x3a 0x0.
14:13:32:24.425 cpu6:4195)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe: NMP device
"naa.60060160432017005c97aea1b32fdc11" state in doubt; requesting fast path state update...
14:13:32:24.425 cpu6:4195)ScsiDeviceIO: 746: Command 0x2a to device
"naa.60060160432017005c97aea1b32fdc11" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x2 0x3a 0x0.
14:13:32:26.718 cpu4:4198)<3> rport-4:0-0: blocked FC remote port time out: saving binding
14:13:32:26.718 cpu4:4198)<3> rport-4:0-1: blocked FC remote port time out: saving binding
14:13:32:26.718 cpu5:4101)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x410004286980) to NMP
device "naa.60060160432017005c97aea1b32fdc11" failed on physical path "vmhba1:C0:T0:L1" H:0x1 D:0x0
P:0x0 Possible sense data: 0x0 0x0 0x0.
14:13:32:26.718 cpu5:4101)WARNING: NMP: nmp_DeviceRetryCommand: Device
"naa.60060160432017005c97aea1b32fdc11": awaiting fast path state update for failover with I/O
blocked...
14:13:32:26.718 cpu5:4101)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x4100042423c0) to NMP
device "naa.60060160432017005c97aea1b32fdc11" failed on physical path "vmhba1:C0:T0:L1" H:0x1 D:0x0
P:0x0 Possible sense data: 0x0 0x0 0x0.
14:13:32:26.718 cpu3:4281)WARNING: VMW_SATP_CX: satp_cx_otherSPIsHung: Path "vmhba1:C0:T1:L1" MODE
SENSE PEER SP command failed 0/1 0x0 0x0 0x0.
14:13:32:26.719 cpu1:4206)WARNING: NMP: nmp_DeviceAttemptFailover: Retry world failover device
"naa.60060160432017005c97aea1b32fdc11" - issuing command 0x410004286980
14:13:32:26.752 cpu2:4237)NMP: nmp_CompleteRetryForPath: Retry world recovered device
"naa.60060160432017005c97aea1b32fdc11"
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
Messages Log (VMware ESX 4.0)
Jul 24 19:13:33 giza sshd[18915]: Connection from 10.16.112.24 port 1396
Jul 24 19:13:36 giza sshd[18915]: Accepted password for root from 10.16.112.24 port
1396 ssh2
Jul 24 19:13:36 giza sshd(pam_unix)[18915]: session opened for user root by (uid=0)
Jul 29 01:01:03 giza iscsid[32725]: cannot make connection to 10.16.95.161:3260: No
route to host
Console events
Logon events
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
hostd.log
Located in /var/log/vmware
Sym-linked to the current rotated hostd log file
Hostd events
VI Client communications when directly connected to
VMware ESX
Events done on behalf of
VPXA
System Services
Firewall System
HA services
VMware Converter
vm-support collects hostd-support*.tgz files
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
hostd-support dump
vm-support collects hostd-support*.tgz files located
in /var/log/vmware directory
Generated by /usr/sbin/vmware-hostd-support script
Rotated up-to 4 copies (blank, -1, -2 and -3 suffix)
Extracted to hostd-support-<pid> directory which includes
Config
Content of /etc/vmware/hostd directory
Logs
/var/log/vmware/hostd-*.log file with current hostd’s PID
/var/log/vmware/hostd-trace.log file if exists
Runtime
Content of /var/lib/vmware/hostd/journal directory
Content of /var/lib/vmware/hostd/stats directory
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
hostd.log
Log for VMware ESX, pid=3049, version=4.0.0, build=build-140815, option=Release,
section=10
Compression of /var/log/vmware//hostd-3.log to /var/log/vmware//hostd-3.log.gz took 167 ms
[2009-04-27 08:39:13.157 F624BB90 info 'ha-eventmgr'] Event 3558 : Lost access to volume
492c94b9-7b4f6ee8-bb0b-001a645dba02 (PSE-i144-100gb) due to connectivity issues.
Recovery attempt is in progress and outcome will be reported shortly.
[2009-04-27 08:36:14.927 F56EBB90 info 'ha-eventmgr'] Event 3549 : Successfully restored
access to volume 492c94b9-7b4f6ee8-bb0b-001a645dba02 (PSE-i144-100gb) following
connectivity issues.
[2009-04-27 08:36:15.307 F6188B90 verbose 'DvsTracker'] FetchSwitches: added 0 items
[2009-04-27 08:36:15.307 F6188B90 verbose 'DvsTracker'] FetchDVPortgroups: added 0 items
[2009-04-27 08:36:34.081 F61C9B90 verbose 'PropertyProvider'] RecordOp 1:
latestPage[3540], session[524622d3-a4f2-0704-e090-fedc59dbc2f5]52553360-a9c2-63b0-7138-
0486ae7d4b0f
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
oldconf Files
esx.conf.2009-03-11_16:39:06 esx.conf.2009-04-14_10:01:57
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.