ESXTOP Guide

The esxtop utility provides a detailed look at how ESXi uses resources in real time. It is
probably the most useful utility to troubleshoot performance issues on an ESXi host. The
tool is available either on the ESXi host itself (#esxtop), or from remote systems like the
vMA (#resxtop).
In vSphere 6.5, esxtop has 9 panes that can be enabled with the following shortcuts:

o c: CPU
o m: Memory
o v: Disk Virtual Machine
o u: Disk Device
o d: Disk Adapter
o n: Network
o i: Interrupt
o p: Power Management
o x: vSAN

CPU Panel
Key Metric Name Description
A ID ID The ID of a running world. The world ID is
typically cloaked and the group ID is displayed,
unless the group is expanded with the "e" command.
This also applies to groups with a single world. The
ID is never identical to the GID.
B GID Group ID The Group ID of a running world. A group is
sometimes also referred as Resource Pool which has
nothing in common with the Resource Pools you
can configure in a DRS Cluster.

Press 'N' to sort by GID column.

C LWID Leader World The Leader World ID, also known as VMX Cartel
ID ID for Virtual Machines. The LWID typcally the
first world that has been started in a group.
D NAME Name The name of a World's group. The number
appended to the name is the LWID. When the group
Key Metric Name Description
is expanded with the "e" command the name of the
World is displayed.
E NWLD Number of The number of worlds running in a Group. NWLD
Worlds is 1 when either the Group as a single process, or
when the group has been expanded with the "e"
command.
F %USED Used Percentage of physical CPU core cycles used by the
world or group. %USED depends on the frequency
the CPU core is running with and can be higher or
lower compared to %RUN when the nominal (rated)
frequency differs. Groups can also be higher than
100% when more vCPUs are configured or there is
a high %SYS usage.
%USED = %RUN + %SYS - %OVRLP

Press 'U' to sort by %USED column.

F %RUN Run Percentage of total scheduled time for the world to
run. %RUN does not account hyperthreading and
system time. On a hyperthreading enabled server,
%RUN can be twice as large as %USED.
The main difference between %RUN and %USED
is that %RUN does not account system time.
F %SYS System Percentage of time spent by the ESXi VMkernel on
behalf of the world to process interrupts and to
perform other system activities. A high %SYS
usually means high IO.
F %WAIT Wait Percentage of time a world spent in the blocked or
busy wait state. This percentage also includes
%IDLE. The theoretical maximum of %WAIT is
(NWLD*100). It's nothing wrong when there is a
high value because the idle time is included in
%WAIT.
100%= %WAIT + %RDY + %CSTP + %RUN
F %VMWAIT Virtual The total percentage of time a world spent in a
Machine blocked state waiting for events. The %VMWAIT
Wait metric is only available for vCPUS or combined in a
VM group.
For vCPUS when a Virtual Machine is expandede
with the "e" command, %WAIT = %VMWAIT +
%IDLE + %SWPWT
F %RDY Ready The percentage of time the world was ready to run
but waiting in a queue to be scheduled. This may
happen even when there is plenty of free CPU
cycles when a VM CPU is administratively limited
Key Metric Name Description
which is shown by %MLMTD.
To determine CPU contention from %RDY you
have to take into account %RDY, %MLMTD and
the number of vCPUS. If %RDY - %MLMTD is
higher than 10% per vCPU you are very likely
experiencing CPU contention. Typically you want
to see %RDY close to 0.
100% = %RUN + %RDY + %CSTP + %WAIT

Press 'R' to sort by %RDY column.

F %IDLE Idle The percentage of time the vCPU world is in an idle
loop. %IDLE is only available for vCPU worlds.
Other worlds do not have idle loops, so %IDLE is 0.
F %OVRLP Overlap The percentage of time spent by system services on
behalf of other worlds. For example, if VM A is
currently being scheduled and a network packet for
VM B is processed by the ESXi VMkernel, the time
spent appears as %OVRLP for virtual machine A
and %SYS for virtual machine B.
F %CSTP CoStop The percentage of time that the VM is ready to run
but is waiting for the availability of other vCPUs.
The co-deschedule state applies only for SMP VMs.
The CPU scheduler might put a vCPU in this state,
when the VM workload does not use vCPUs in a
balanced fashion. For example, if you have a VM
with 2 vCPUs running a non SMP aware
application, utilizing 1 vCPU at 100% and 1 vCPU
at 0%. In that case, the CPU scheduler penalties the
VM to reduce resource shortage for other VMs.
This is is represented as %CSTP.
F %MLMTD Max Limited The percentage of time a world was ready to run but
deliberately wasn't scheduled because doing so
would violate the resource pool or virtual machine
CPU limit. The %MLMTD time is included in
%RDY time. If the value is not 0, the VM has been
administratively limited.
F %SWPWT Swap Wait The percentage of time a world spends waiting for
the VMkernel to swap memory. The %SWPWT
time is included in the %WAIT time. If %SWPWT
is greater than 0, VMkernel is swapping the VMs
memory to the disk. This will have a large negative
impact in overall performance. Swapping can be
caused by high memory oversubscription or
memory limits configured on a resource pool or
VM.
Key Metric Name Description
G SWTCH/s Switches/sec Number of world switches per second. A context
switch occurs when a CPU switches the execution
from one world to another.
G MIG/s Migrates/sec Number of migrations per second. A world can be
migrated from a busy pCPU to a less loaded pCPU
for load balancing.
G QEXP/s Quantum Number of quantum expirations per second. This
Expires/sec happens when a time quantum given to a currently
running world expires. It is common that a world
changes its state to WAIT before the current time
quantum expires (50ms).
G WAKE/s Wakeups/sec Number of world wakeups per second. A world
wakes up when its state is changes from WAIT to
READY
H AMIN Alloc Min Resource pool, virtual machine, or world attribute
reservation. A value of 0 means no reservation.
H AMAX Alloc Max Resource pool, virtual machine, or world attribute
limit. A value of -1 means unlimited.
H ASHRS Alloc Shares Resource pool, virtual machine, or world attribute
Shares. Default shares are -4 (High), -3 (Normal)
and -2 (Low). When a custom share is configured,
the value is displayed.
H AMLMT Alloc Min Undocumented parameter.
Limited
H AUNITS Allocated Allocation unit (MHz for virtual machines)
unit
I %LAT_C Latency CPU The percentage of time the Resource Pool/World
was ready to run but was not scheduled to run
because of cpu resource contention.
I %LAT_M Latency The percentage of time the Resource Pool/World
Memory was ready to run but was not scheduled to run
because of memory resource contention.
I %DMD Demand The CPU demand in percentage. It represents the
average active CPU load in the past 1 minute.
I EMIN Effective Min Effective minimum cpu allocation in case of
resource contention. This is the amount of MHz the
world will receieve when virtual machines are
competing for resources.
I TIMER/s Timer/sec Timer rate for this world.
I AFFINITY_BIT_MASK Affinity Bit Bit mask showing the current scheduling affinity for
Mask the world.
I CPU CPU The physical or logical processor on which the
world was running when resxtop (or esxtop)
obtained this information.
Key Metric Name Description
I HTSHARING HT Sharing Current hyperthreading configuration.
I HTQ HT Indicates whether the world is currently quarantined
Quarantine or not. N means no and Y means yes.
J POWER Power Usage Current CPU power consumption for a virtual
Watts machine (in Watts). Usage calculation is based on
the value provided by the power supply.

Memory Panel
Key Metric Name Description
A ID ID The actual world ID is cloaked and the group ID is
displayed. As you can't expand the group in the memory
screen you can't see the real world ID.
B GID Group ID The Group ID of a running world. A group is sometimes
also referred as Resource Pool which has nothing in
common with the Resource Pools you can configure in a
DRS Cluster.

Press 'N' to sort by GID column.

C LWID Leader World ID The Leader World ID, also known as VMX Cartel ID for
Virtual Machines. The LWID typcally the first world that
has been started in a group.
D NAME Name The name of a World's group. The number appended to the
name is the LWID. Virtual Machines do not have a number
appended.
E NWLD Number of The number of worlds running in the Group. NWLD is 1
Worlds when the Group as a single process.
F AMIN Alloc Min Resource pool, virtual machine, or world attribute
reservation. A value of 0 means no reservation.
F AMAX Alloc Max Resource pool, virtual machine, or world attribute limit. A
value of -1 means unlimited.
F ASHRS Alloc Shares Resource pool, virtual machine, or world attribute Shares.
Default shares are -4 (High), -3 (Normal) and -2 (Low).
When a custom share is configured, the value is displayed.
F AMLMT Alloc Min Undocumented parameter.
Limited
F AUNITS Allocated unit Allocation unit (kilobytes for virtual machines)
G NHN Numa Home Current home node for the resource pool or virtual
Nodes machine. This statistic is applicable only on NUMA
systems. If the virtual machine has no home node, a dash (-
) appears. A virtual machine runs only on processors within
its home node, and its newly allocated memory comes from
the home node as well. A virtual machine can have
Key Metric Name Description
multiple home nodes when the number of virtual CPUs
exceeds the number of cores per physical socket.
G NMIG Numa Rebalance
Count Delta
G NRMEM Numa Remote Amount of remote memory allocated to the virtual
Memory MBytes machine.
G NLMEM Numa Local Amount of local memory allocated to the virtual machine.
Memory MBytes
G N%L Numa % Local Percentage of local memory allocated to the virtual
machine. This value should be close to 100%. A reason for
poor NUMA locality can be that a virtual machine has
more memory configured than it is available for each
processor. Accessing remote memory causes increased
latency.
H MEMSZ Memory Size Amount of physical memory allocated to a virtual machine.
MBytes This is the configured virtual machine memory.
MEMSZ = GRANT + MCTLSZ + SWCUR + untouched

Press 'M' to sort by MEMSZ column.

H GRANT Memory Granted Amount of physical memory mapped to the virtual
Size MBytes machine. If GRANT is lower than MEMSZ, either the
guest has never used all of its configured memory or if it
has been reclaimed by the balloon driver.
H CNSM Memory Amount of the memory consumed by the virtual machine.
Consumed Size The memory currently consumed by the virtual machine is
MBytes equal to the amount of memory that the VM guest
operating system currently uses, excluding the amount of
memory saved by transparent page sharing and memory
compression.
H SZTGT Target Size Targeted amount of physical memory to be allocated. It is
MBytes possible that SZTGT is higher than MEMSZ because it
includes the virtual machine overhead memory.
H TCHD Touched MBytes The amount of physical memory recently used (read or
write) by the virtual machine. Touched memory is a
working set estimate, which indicates how actively the VM
is using its memory. This value is similar to the active
memory reported by the guest OS.
H TCHD_W Touched Write The amount of physical memory recently written by the
MBytes virtual machine.
I %ACTV Active Estimate Percentage of active guest physical memory, current value.
I %ACTVS Active Slow Percentage of active guest physical memory, slow moving
Estimate average.
I %ACTVF Active Fast Percentage of active guest physical memory, fast moving
Estimate average.
Key Metric Name Description
I %ACTVN Active Next Percentage of active guest physical memory, predict of
Estimate what %ACTVF will be at next sample.
J MCTL? Memctl? Memory balloon driver is installed or not. N means no, Y
means yes. The ballooning driver is part of VMware Tools.
J MCTLSZ Memctl MBytes Amount of physical memory reclaimed from the virtual
machine by ballooning. To decrease host memory pressure,
the ballon driver inflates inside the virtual machine and
makes the physical memory available for other virtual
machines. The performance impact caused by ballooning is
small and therefore preferred over swapping. Another
reason for ballooning can be a configured memory limit
(AMAX).

Press 'B' to sort by MCTLSZ column.

J MCTLTGT Memctl Target Amount of physical memory the ESXi system attempts to
MBytes reclaim from the virtual machine by way of ballooning.
J MCTLMAX Memctl Max The maximum amount of guest physical memory
MBytes reclaimable by the balloon driver.
K SWCUR Swapped MBytes Current swap usage by this virtual machine.
K SWTGT Swap Target Target where the ESXi host expects the swap usage by the
MBytes virtual machine to be.
K SWR/s Swap Read Rate at which the ESXi host swaps in memory from disk
MBytes/sec for the virtual machine.
K SWW/s Swap Written Rate at which the ESXi host swaps virtual machine
MBytes/sec memory to disk.
L LLSWR/s Llswap Read The rate at which memory is read from the host cache.
MBytes/sec When host caching is enabled, the ESXi host is able to
swap to a local SSD, instead of the virtual machines
datastore, which significantly reduces the impact caused by
swapping.
L LLSWW/s Llswap Written Rate at which memory is written to the host cache. When
MBytes/sec host caching is enabled, the ESXi host is able to swap to a
local SSD, instead of the virtual machines datastore, which
significantly reduces the impact caused by swapping.
M CPTRD Checkpoint Read The amount of data read from the checkpoint file. Data
MBytes from a checkpoint file is read when a virtual machine is
resumed from a suspended state.
M CPTTGT Checkpoint The size of checkpoint file. The checkpoint file is used
Target MBytes when a virtual machine is suspended.
N ZERO Zero MBytes The size of virtual machine physical pages that are zeroed.
A zero page is simply a memory page that is all zeros
which can be easily used for page sharing.
N SHRD Shared MBytes Amount of guest physical memory that is shared.
Key Metric Name Description
N SHRDSVD Shared Saved Amount of physical memory that is saved due to page
MBytes sharing.
N COWH Copy On Write Amount of guest physical hint pages for page sharing.
Hint MBytes
O OVHDUW Overhead UW Amount of overhead memory reserved for the vmx user
MBytes world.
O OVHD Overhead MBytes Amount of overhead memory currently consumed by the
virtual machine.
O OVHDMAX Overhead Max Amount of reserved overhead memory for the entire VM.
MBytes
P MCMTTGT Min Commit
Target MBytes
P CMTTGT Commit Target
MBytes
P CMTCHRG Commit Charged
MBytes
P CMTPPS Commit Pages
Per Share
Q CACHESZ Compressed Current size of compressed memory by this virtual
Memory MBytes machine.
Q CACHEUSD Used Compressed Used compression memory cache.
Memory MBytes
Q ZIP/s Compression Compressed memory per second.
MBytes/sec
Q UNZIP/s Decompression Decompressed memory per second.
MBytes/sec

Disk (Virtual Machine) Panel

Key Metric Name Description
A ID ID The virtual SCSI Disk ID. The ID is typically cloaked and
the group ID is displayed, unless the group is expanded
with the "e" command. It is the same ID as used by
vscsiStats.
B GID Group ID The Group ID of the virtual machine.
C VMNAME Virtual Machine The display name of the virtual machine.
Name
Press 'N' to sort by VMNAME column.
D VDEVNAME Virtual Device Name of the VSCSI device. Only displayed when the
Name group is expanded with the "e" command.
E NVDISK Number of Number of VSCSI devices.
Virtual Disks
Key Metric Name Description
F CMDS/s Commands/sec Number of commands issued per second.
F READS/s Reads/sec Number of read commands issued per second.

Press 'r' to sort by READS/s column.

F WRITES/s Writes/sec Number of write commands issued per second.

Press 'w' to sort by WRITES/s column.

F MBREAD/s MBytes Read/sec Megabytes read per second.

Press 'R' to sort by MBREAD/s column.

F MBWRTN/s MBytes Megabytes written per second.
Written/sec
Press 'T' to sort by MBWRTN/s column.
G LAT/rd Read Latency Average latency (in milliseconds) per read.
H LAT/wr Write Latency Average latency (in milliseconds) per write.

Disk (Device) Panel

Key Metric Name Description
A DEVICE Device Name of the storage device.
B PATH/WORLD/PARTITION Path/World/Partition This column is only visible when
Id expanded with either "e" for Disk
World Statistics, "P" for Disk Path
Statistics or "t" for Disk Partition
Statistics.
C NPH Number of Paths Number of paths to the storage device.
C NWD Number of Worlds Number of worlds that are running on
the storage device. Expand the device
with the "e" command to display
statistics for individual worlds.
C NPN Number of Partitions Number of partitions on the storage
device.
D SHARES Shares Number of shares. Only displayed
when the device is expanded with the
"e" (Disk World Statistics) command.
E BLKSZ Block Size (Bytes) Physical Device block size in bytes.
E NUMBLKS Number of Blocks Number of blocks of the device.
F DQLEN Device Q Depth Current device queue depth of the
storage device.
F WQLEN World Q Depth World queue depth. This is the
maximum number of ESXi VMkernel
active commands that the world is
allowed to have. This is a per device
Key Metric Name Description
maximum for the world. It is valid
only if the corresponding device is
expanded to worlds.
F ACTV Active Commands Number of commands in the ESXi
VMkernel that are currently active.
This statistic applies to only worlds
and devices.
F QUED Queued Commands Number of commands in the ESXi
VMkernel that are currently queued.
This statistic applies to only worlds
and devices.
F %USD % Used Percentage of the queue depth used by
ESXi VMkernel active commands.
This statistic applies to only worlds
and devices.
F LOAD Load Ratio of ESXi VMkernel active
commands plus ESXi VMkernel
queued commands to queue depth.
This statistic applies to only worlds
and devices.
G CMDS/s Commands/sec Number of commands issued per
second.
G READS/s Reads/sec Number of read commands issued per
second.
G WRITES/s Writes/sec Number of write commands issued per
second.
G MBREAD/s MBytes Read/sec Megabytes read per second.
G MBWRTN/s MBytes Written/sec Megabytes written per second.
H RESV/s Reserves/sec
H CONS/s Conflicts/sec
I DAVG/cmd Average Driver This is the average response time in
MilliSec/Command milliseconds per command being sent
to the device. The DAVG value is part
of GAVG.
I KAVG/cmd Average Kernel This is the amount of time the
MilliSec/Command command spends in the VMkernel.
The KAVG value is part of GAVG.
I GAVG/cmd Average Guest "This is the response time as it is
MilliSec/Command perceived by the guest operating
system. This number is calculated with
the formula: DAVG + KAVG =
GAVG"
I QAVG/cmd Average Queue Average queue latency per command
MilliSec/Command in milliseconds.
Key Metric Name Description
J DAVG/rd Average Driver Average device driver read latency per
MilliSec/Read read operation in milliseconds.
J KAVG/rd Average Kernel Average ESXi VMkernel read latency
MilliSec/Read per read operation in milliseconds.
J GAVG/rd Average Guest Average guest operating system read
MilliSec/Read latency per read operation in
milliseconds.
J QAVG/rd Average Queue Average queue read latency per read
MilliSec/Read operation in milliseconds.
K DAVG/wr Average Driver Average device write latency per write
MilliSec/Write operation in milliseconds.
K KAVG/wr Average Kernel Average ESXi VMkernel write latency
MilliSec/Write per write operation in milliseconds.
K GAVG/wr Average Guest Average guest operating system write
MilliSec/Write latency per write operation in
milliseconds.
K QAVG/wr Average Queue Average queue write latency per write
MilliSec/Write operation in milliseconds.
L FCMDS/s Failed Commands/sec
L FREAD/s Failed Reads/sec
L FWRITE/s Failed Writes/sec
L FMBRD/s Failed Bytes Read/sec
L FMBWR/s Failed Bytes
Written/sec
L FRESV/s Failed Reserves/sec
L ABRTS/s Aborts/sec Number of commands aborted per
second.
L RESETS/s Resets/sec Number of commands reset per
second.
M PAECMD/s PAE Commands/sec Number of PAE commands per
second. This statistic applies to only
paths.
M PAECP/s PAE Copies/sec Number of PAE copies per second.
This statistic applies to only paths.
N SPLTCMD/s Split Commands/sec Number of split commands per
second. This statistic applies to only
paths.
N SPLTCP/s Split Copies/sec Number of split copies per second.
This statistic applies to only paths.
O CLONE_RD VAAI Clone Reads
O CLONE_WR VAAI Clone Writes
O CLONE_F VAAI Clone Failed
Key Metric Name Description
O MBC_RD/s MBytes Clone
Reads/sec
O MBC_WR/s MBytes Clone
Writes/sec
O ATS Atomic Test and Set
O ATSF Atomic Test and Set
Failed
O ZERO Zeros
O ZERO_F Zeros Failed
O MBZERO/s MBytes Zeroed/sec
O DELETE Delete
O DELETE_F Delete Failed
O MBDEL/s Mbytes Delete/sec
P CAVG/suc Average Success
Latency ms/Clone
P CAVG/f Average Failure
Latency ms/Clone
P AAVG/suc Average Success
Latency ms/ATS
P AAVG/f Average Failure
Latency ms/ATS
P ZAVG/suc Average Success
Latency ms/Zero
P ZAVG/f Average Failure
Latency ms/Zero

Disk (Adapter) Panel

Key Metric Name Description
A ADAPTR Adapter
B PATH Path
C NPTH Number of Paths
D AQLEN Adapter Q Depth
E CMDS/s Commands/sec
E READS/s Reads/sec
E WRITES/s Writes/sec
E MBREAD/s MBytes Read/sec
E MBWRTN/s MBytes Written/sec
F RESV/s Reserves/sec
F CONS/s Conflicts/sec
G DAVG/cmd Average Driver MilliSec/Command
Key Metric Name Description
G KAVG/cmd Average Kernel MilliSec/Command
G GAVG/cmd Average Guest MilliSec/Command
G QAVG/cmd Average Queue MilliSec/Command
H DAVG/rd Average Driver MilliSec/Read
H KAVG/rd Average Kernel MilliSec/Read
H GAVG/rd Average Guest MilliSec/Read
H QAVG/rd Average Queue MilliSec/Read
I DAVG/wr Average Driver MilliSec/Write
I KAVG/wr Average Kernel MilliSec/Write
I GAVG/wr Average Guest MilliSec/Write
I QAVG/wr Average Queue MilliSec/Write
J FCMDS/s Failed Commands/sec
J FREAD/s Failed Reads/sec
J FWRITE/s Failed Writes/sec
J FMBRD/s Failed Bytes Read/sec
J FMBWR/s Failed Bytes Written/sec
J FRESV/s Failed Reserves/sec
J ABRTS/s Aborts/sec
J RESETS/s Resets/sec
K PAECMD/s PAE Commands/sec
K PAECP/s PAE Copies/sec
L SPLTCMD/s Split Commands/sec
L SPLTCP/s Split Copies/sec

Network Panel
Key Metric Name Description
A PORT-ID Port ID The port ID used on the virtual switch. This ID is used
for example for network traces with pktcap-uw.

Press 'N' to sort by PORT-ID column.

B UPLINK Uplink? This virtual switch port is a physical uplink port. N
means no, Y means yes.
C UP Link Up? Y means the corresponding link is up. N means it is
not. Only visible for uplink ports.
C SPEED Link Speed (Mb/s) Link speed in Megabits per second. Only visible for
uplink ports.
C FDUPLX Full Duplex? Y means the corresponding link is operating at full
duplex. N means it is not. Only visible for uplink
ports.
Key Metric Name Description
D USED-BY Used By Indicates what is connected to the virtual switch port.
Connected devices can be virtual machine NICs,
VMkernel ports (vmk#), physical NICs (vmnic#) or
ports used for health checks (Shadow of vmnic#).
E TEAM-PNIC Team Uplink The physical NIC that the corresponding device is
Physical NIC Name actively using. This information is helpful for network
troubleshooting.
F DNAME Device Name Virtual Switch name. Either the display name for
virtual switches or DvsPortset-# for distributed
switches.
G PKTTX/s Packets Number of packets transmitted per second.
Transmitted/sec
Press 't' to sort by PKTTX/s column.
H MbTX/s MBits MegaBits transmitted per second.
Transmitted/sec
Press 'T' to sort by MbTX/s column.
I PSZTX Average Packet Size Average size of transmitted packets in bytes.
Transmitted (Bytes)
J PKTRX/s Packets Number of packets received per second.
Received/sec
Press 'r' to sort by PKTRX/s column.
K MbRX/s MBits Received/sec MegaBits received per second.

Press 'R' to sort by MbRX/s column.

L PSZRX Average Packet Size Average size of received packets in bytes.
Received (Bytes)
M %DRPTX % Outbound Packets Percentage of transmit packets dropped.
Dropped
N %DRPRX % Received Packets Percentage of receive packets dropped.
Dropped
O ACTN/s Actions Posted/sec Number of Vmkernel actions posted per second. This
is an VMware internal counter with no further
documentation.
P PKTTXMUL/s Multicast Packets Number of multicast packets transmitted per second.
Transmitted/sec
P PKTRXMUL/s Multicast Packets Number of multicast packets received per second.
Received/sec
Q PKTTXBRD/s Broadcast Packets Number of broadcast packets transmitted per second.
Transmitted/sec
Q PKTRXBRD/s Broadcast Packets Number of broadcast packets received per second.
Received/sec