# options 100 ©100_speed_duplex=2 # options ©1000 Speed=[0,10,100,1000 Duplex: # options acenic link=[0x213, 0x212,0x223, 0x222,0x243,0x242,0x271,0%270 # options 3c990 force=[0..4] Notice the options available for network cards in this file, If we cannot use the mii-tool to force NIC speed and duplex, then we can remove the comment character ("") from the appropriate options line in the this file, If there are problems with the interface eth0 disappearing after a rescan SAN operation, ensure that the Ethernet alias definitions above eth0 (i.e. eth1, eth2 etc.) in modules.conf are commented out. modinfo ‘This tool takes a service console driver and displays the options it supports. For exemple modinfo 1000 Would produce a list of flow control settings for the Intel gigabit NIC. modinfo cciss ‘Would produce the file details and version of the HP Smart Array controller. insmod Insert module. This command loads 2 device driver module, ‘You are only likely to encounter this command if you decide to share your service console physical network ‘ard with the VMkernel, when it is used to load the vmxnet_console device driver module. modprobe /etc/vmware/vmkmodule.conf ‘This file lists the device driver modules that the VMkernel will load. This is the VMkernel equivalent to the service console modules.con* file. Notice that it is vnknodute.conf and not plural, as is the case with the equivalent service console file! ‘The VMkernel device driver modules themselves are stored in /usr/Lib/vmware/vmkmod megaraid.o nfshaper.o tepip qla2200_604.0 Ispci Great tool for listing pci devices. Could be used to demo what the VM is presenting to the guest OS.Aspei -v verbose output spei -t print tree of PCI devices (use tv for verbose tree) ‘You may wish to examine /proc/pci also in order to correctly identify PCI devices and their slot configurations. One point to note is that when you are faced with PCI slot numbers is that not all hardware vendors number their slots in a straight forward left to right configuration. Make sure you know your slot numbers and their layout! Isusb {A tool to list USB devices. Gives out way more info that is actually required. Remember that USB devices cannot be presented to virtual machines in ESX Server. If you wish to use a USB device in ESX, then you will have to use a USB over IP device and install the appropriate driver software into your guest OS for this. The most common USB over IP dovice is AnywhereUSB from Digi, Details can be found at www.digi comMpraducts/ush, A company called Keyspan also produce a similar device, details at seww-keyspan.com kudzu ‘A Red Hat tool to detect and configure hardware. However, be careful using this tool with VMware ESX if you are making changes to network or HBA PCI resources. Jetc/vmware/netmap.conf This text file maps ESX virtual switch names to device names. It is a network map configuration file as ‘opposed to the more generic devices map config file (devnames.conf) network0.name = "SecuredGigabi network0.device = "vmnic2" networkl.name = "Virtualswitch1" networkl.device = "vmnet_1" network2.name = "InternetSwitch" network2.device = "vmnicl" Remember that Ifa virtual switch has no physical adapters, then it is vmnet. Ifa virtual switch has only 1 physical adapter it is a vmnic Ifa virtual switch has two or more physical adapters, it is a bond. ‘A bond can be in one of three modes, out-mac (default), out-ip and standby out-mac A VM virtual NIC is assigned to a pNIC in the bond and it uses only that out-ip A VM TCP conversation is placed on an available pNIC standby AVM will only use one NIC until a failure, then the other is used, There is no point in having morethat 2 NICs in a bond in this mode. /etc/vmware/devnames.conf ‘This text file maps device names (example above) to modules and their PCI addresses. Note that the devnanes..cont file contains SCSI devices and NIC devices. megaraid vmhba0 nic vmnic0 nic vmnicl nic vmnic2 (/etc/vmware/vmware-devices.map ‘Appears to be like a hardware compatibility list. Watch out for creating your own device map, devices.local The /etc/vmware/vmware-devices.map file contains 2 list of devices supported by ESX Server. This release includes support for a local version of this file, /etc/vmware/vmware- devices.map.local. Modify the vmware-devices.map.local to select different device drivers. This file is not modified during an ESX Server upgrade, preserving your customizations. The vmware- devices.map.local is read when the VMkernel is loaded: ‘Any changes to the vmware-devices.map local file require a reboot, or at least an unload/reload of the \VMkernel to take effect, Entries in the vmware-devices.map.local files are used in addition to the entries in the vmware-devices.map file. The vmware-devices.map.local file does not need to mirror the vmware-devices.map file. ‘Any vmware-devices.map local file entries that correspond to the vmware-devices.map file entries supercede the vmware-devices.map file entries Jetc/vmware/config Contains some useful settings. Libdir = "/usr/lib/vmware" dhepd.fullpath = "/usr/bin/vmnet-dhepa" authd.fullpath = "/usr/sbin/vmware-autha" authd.client.port = "90 loop. fullpath = "/usr/bin/vmware-loop" ullpath = "/usr/bin/vmware" -fullpath = "/usr/bin/vmware-cma" -fullpath = "/usr/sbin/vmware-serverd" ullpath = “/usr/bin/vmware-wizard" -init.fullpath = "/usr/lib/vmware/serverd/init.p1" serverd.vpxuser ~ "vpxuser" serverd.snmpdconf.fullpath = "/e snmp.enable = "TRUE" prefvmx.useRecommendedLockedMemsize =autoStart.defaultStartDelay = "240" If this file is missing or corrupted then you will get some very weird behaviour, for example the inability to power on any virtual machine and no vmware . Log file being created. If the root file system should become full, certain files can be corrupted, make sure this isn't one of them. I've seen this file truncated a number of times. To check if this may be your problem, try vmware-cmd /home/vmware/vm/vm.vmx start ‘And if the result is formed protocol error then it could be that the /etc/vmware/config file is truncated, corrupt or simply missing. Normally, simply copying this file from another server will normally restore the server to normal operations, (/etc/vmware/hwconfig Loads of information in this text file. Useful for finding which nic is in which team. If using alongside devnames.conf and netmap.conf use the following command rep -i vmnic /etc/vmware/hwco! /dev/tdo How to address the floppy disk drive. /dev/sda Denotes a SCSI device in the service console. So the first SCSI disk would be sda, the second would be scib and so on. /proc/vmware/pci ‘Text file with discovered PCI devices Bus:S1.F Vend:Dvid Subv:Subd Type Vendor ISA/irg/Vec P M Module Name Spawned bus 000:00.0 8086:3590 1028:016e Host/PCI Intel C 000:02.0 8086:3595 0000:0000 PCI/PCI oo1c 000:03.0 8086:3596 0000:0000 PCT/PCT 004 c 000:04.0 8086:3597 0000:0000 PCI/PCI 007 c 000:05.0 8086:3598 0000:0000 PCI/PCI o10 c 000:06.0 8086:3599 0000:0000 PCT/PCT o13.c 000:29.0 8086:24d2 1028:016e USB I) 16/0x69 AC 000:29.1 8086:24d4 1028:016e USB Intel 10/ 19/0x71 BC 000:29.2 8086:24d7 1028:016e USB Intel 7/ 18/0x79 CC 000:29.7 8086:24dd 1028:016e USB Intel 3/ 23/0x81 DC8086:244e 8086:24d0 8086:24db 8086:0330 PCI/PCI Intel 016 C PCI/ISA Intel ¢ IDE PCI/PCI Inte. 8086:0329 8086:03: 8086:1076 8086:1076 1002:5159 PCI/PCI 0 PCI/PCI Intel 0 E 1 002 8086:0332 PCI/PCI Intel 003 ¢ 1028:0013 RAID Dell 7/ 38/0x91 vmhbad 8086:0329 PCI/PCI Intel 005 ¢ 8086:032a PCI/PCI Intel 006 C 8086:1010 Ethernet 1/ 58/0x99 AC 8086:1010 E 10/ 59/0xal B V e1000 vmnicd ql 12 ernet In 12/0xa9 AV e1000 vmnicl el 11/ 73/0xb1 A V €1000 vmnic2 Display ATI 7/ 18/0x79 AC findnic ‘This tool is also known as “The VMkernel Network Card Locator”. It locates a physical NIC in an ESX server by using pings. It is left to the operator to check which NIC is sending the echo requests by either unplugging network cables or inspecting flickering lights on the NIC or the switch. The useful feature of this tool is we can ping based on the device name that the VMkernel uses to access the NIC. We should remember that each physical NIC in an ESX server that is allocated to the VMkernel does not itself have an IP address, therefore in to perform an ICMP echo request, we need to temporarily give that NIC an IP; this is specified as the first IP parameter in the command, the second IP parameter being the ICMP destination. findnic -i 5 vmnic2 10.0.0.1 192.168.1.3 ‘The above command will send ICMP echo requests to 192.168.1.3 every 5 seconds. We could also use the -£ switch which would flood ping. The /proc Hierarchy /proc ‘The volatile /proc directory hierarchy that can be treated as a file system but is actually held in RAM We can interrogate the files and directories in /proc to find out some great information about the running of the service console. /proc/vmware ‘The volatile /proc/vmware directory hierarchy that can be treated as @ file system but is held in RAM. We can interrogate the files and directories in /proc/vmware to find out some great information about the running of the VMKernel /proc/vmware/sched/cpu'A text file snapshot of CPU scheduling. If you cat this file you can gather some very useful information, Including which CPU a world is running on, the processor affinity, cpu min & max values, shares, runtimes etc. This information is presented in tabular format which can be great for comparing what resources different VMs are running with. The sample output shown below has been modified in width to fit onto this page. ype name uptime s consoze 173599.074 RUN 3576.903 0 0 1 179599.074 RUN 85557.799 2 95198.823 2 2 293.897 3 19599.074 READY 88689.586 9 279599.074 Wa us usedsec cpu affinity wi. 9599.074 Wi +0023: 0,1,2,3 79599.074 WAIT 2.595 1 0,1,2,3 179589.704 WAIT OL 0,1,2,3 41 § driver 179583.358 WAIT 0 0 0,1,2,3 42. N migServer 176083.967 WATT 3.246 | 2 0,1,2,3 52 1s ¥ 671 READY 1299.122 | 0 0,1,2,3 158 158 v RUN 869.384 2 0,1,2,3 If you determined that one of these VMs temporarily needed more CPU shares, you could increase CPU shares on the fly at the command line, just by using echo to input a value into the shares file for that VM. 10000 > /proc/vnware/vm/ When we use the tool esxtop we get presented with two columns that look identical, VCPUID and WID. The VCPUID is the ID number of the virtual processor of that VM. This number will be the same as the world ID of the VMM, indicated in the WID column, Where things get interesting is when we have a VM with 2 virtual CPUs, i.c. we are using virtual SMP (symmetric multiprocessing). In this case, a VM gets two \VCPUIDs, but is still only 1 world, So the output of esxtop when you have a vSMP VM would be similar to idle idle 47. 0.00 0.00 idle 37. 0.00 0.00 idle 34. 0.00 0.00 145° vmm 8. 3.85 7.00 127 console 7. 3.53 9.00 160 vm 5 1.61 4.00 2 0.50 1.00 2. 0.14 0.00 note world ID of A note world ID of In the above example, it can be seen that there are two VCPUIDS (164 & 165) that correspond to the same world ID (164). /proc/vmware/vm/xxx/disk/vmhba You can obtain disk queuing activity from this file for each VM. However, you will likely find that the Width of the data produced is too wide for your screen and you get a horrible line-wrap. ‘To avoid this and view the data in a more sensible fashion, pipe the output of the command into the less ‘command with the -S switch as shown: /proc/vmware/vm/133/disk/vmhbad: It might also be a good idea to use the watch command on this file, as the disk queue length will be constantly changing and when you cat the file, you may only be sampling the queue while its on zero!/proc/vmware/scsi/ ‘A directory which contains subdirectories for each host bus adapter (hba). /proc/vmware/net ‘To view the status of the virtual Ethernet switches in vmkernel and obtain bandwidth measurements, we can Inspect or sample values from the /proc/vnware/net directory. The subdirectories of /proc/vmware/net will correspond to each virtual Ethernet switch defined in the VMkernel ‘These directories are labelled using names vnic, vmnet and bond. If you wish to reconcile a vmnic number to the virtual Ethemet switch name exposed in the MUI, then inspect /et.c/vmware/netmap. conf. In the subdirectories of cach virtual switch (e.g. /proc/vmware/net./vmaicd ) you will find files that correspond to per-virtual MAC address of each VM attached to that VM. File Systems & Mounting ‘smbclient ‘This is a redirector type tool to view and connect to SMB (Microsoft networking) hosts. Before updating the /ete/tstab file with remote file system information, check first using smizc’ient that the share is visible. The following was produced with smbclient -I 192.168.115 - Sharename Type Comment Disk Default share rosoft Disk IPCS IPC Remote IPC Ds Disk Default share NETLOGON Disk Logon server share REMINST Disk Remote Installation ADMINS Disk Remote Admin SYSVOL | Disk Logon server share \VPLOGON Disk Symantec AntiVirus cs Disk Default share VPHOME Disk Symantec AntiVirus ‘To create a mount point to a Microsoft share is very straightforward, Remember, we are allowing the servi console to access a remote file system. This is not related to what virtual machines are doing. Further, we need to be careful if we are attempting to do any file operations due to potential limits with 2GB file sizes. 1. Add entry to local hosts (/ete/hosts) for MS host 2. Create a local directory where you wish the mount point with mkcix 3. Run smbclient -I -U user -L NetBIOSComputername to check you can sce the shares4, Add an entry to /ete/fstab //server/share /mountdir smbfs ip=ipaddress, username=user, pass jord=pass,noauto 0 0 '5, Mount the remote file system with mount /mountdir ‘6. Change directory into mount point and the directory will no longer be empty but will appear just like a mapped drive. Altematively, if you just want to map temporarily to a Microsoft host and not have to modify fstab, then use ‘smbelient interactively as shown: smbclient //ipaddress/share -U N' \NTuser Password: **#*## Domain=[TAUPO] OS=[Windows 5.0] Server=[Windows 2000 LAN Manager] smb: \> NFS Network File System (NFS) is provided by =pc.nfsd and would normally be launched by an nfs script in /etc/rc.d. To start using NFS to mount directories on other ESX or Linux servers, we can use the following steps: 1, Change the ESX Server which is to be the NFS server, to use medium security (using the MUI is for this). 2, Add an entry to the file /etc/exports on the NFS or use the exportfs command server by either using vi text editor (/vmimages *) 3. Check that the NFS client IP address or hostname is not excluded by the server file /etc/hosts.deny 4 It is up to you if you explicitly allow the NFS client by adding the NFS client to /etc/hosts.allow on the NFS server 5, Start the NFS server daemons a, /ete/re.d/init.d/service portmap start b, /ete/re.d/init.d/service nfslock start c, /ete/re.d/init.d/service nfs st 6. Start the NFS client daemons a. /etc/re.d/init.d/service netfs start b. /ete/re.d/init.d/service portmap start 17. Mount remote export directory (on the NFS server) from the NFS client with the command a,mount -t nfs nfsserver:/export localdir/localmount showmount ‘This command is used by a NFS client to see what directories are being exported by a NFS server.showmount -e nfsserver ‘This command can be specified with the hostname name or IP address of the NFS server holding the exported directories, exportfs ‘The export £s command allows you to selectively export or unexport directories without restarting the various NFS services. nfsconfig NIS NIS stands for Network Information Service. This was formerly called Yellow Pages (YP). ‘Amongst other things, NIS can ensure that the numeric user IDs are unique across the organisation. This is because numeric user IDs are used in NFS, so we can have a mistaken identity situation as user id 515 on a nfs client will not be the same as user id 515 on a nfs server. vmware-mount.pI Just like the Windows utility to mount virtual disks when they are powered off so you can check what's in them. This will mount ext3 and vfat as read/write but NTFS as read only. To find out what file systems are in the virtual disk, use the ~p switch. ymware-mount.pl -p /vmfs/VMFS-VoL1/win2k3.vmak VMware for Linux - Virtual Hard Disk Mout Copyright 1998 VMware, Inc. All rights ri ter Version: 1.0 build-9638 served. -- VMware Confidential Size Type Id sytem 8160957 BIOS 7 HPFS/NIFS Ifwe actually want to mount a partition then we need to be specific and create a directory (or use an existing) to be our mount point. inkdir /myntfs ymware-mount .pl /vmfs/VMFS-VOL1/win2k3.vmdk 1 fs -o ro /myntfs ‘This command will tie up the console window hence you'll need to spawn a new window first to navigate to /myntfs to view the contents. mount Mount file system This command can be used with the -t switch to specify file system type, e.g. nfs, smbfs or iso9660mount -t nfs [-o options] esx4:/vmimages /root/other_server_vmimages ‘To mount a ed, you could just use mount /mnt/cdrom For defined mount points we use the file /ete/fstab, We can also use mount to gain access at the service console to an ISO image or floppy disk image using the following mount syntax: mount -o loop cdname.iso mount-dire ory Ifyou want to make your own ISOs then you can use the utility mxisof's (not included in the service console) to select the files and create the ISO file, then use the cdrecord utility to write to device. umount Un-mount file system. Note it is u-mount and not unmount! smbmount Amount. ~t smb£s passes control across to this utility. We can use this utility directly if we prefer for mounting SMB host file systems. /etc/fstab ‘This is the file system table. This file describes the partitions and storage that the service console can access and how. The first column is the device name, the second is the mount point. LABEL=/ / LABEL=/boot. /boot none /dev/pts oo LABEL=/home /home none /proc none /dev/shm LABEL=/vmimages /vmimages ext3 /dov/sda3 swap swap /dev/cdrom /mnt/cdrom iso9660 er,kudzu,ro 0 0 /dev/ £a0 /mnt/floppy a ner, kudzu 0 0 //taupodcl/msfiles /root/share smb: p=192.168.1.150,credentials=/root/.smbcreds,noauto 0 0 If we have smb mount points defined in the fstab file, then this file could end up with user credentials in it. The fstab file is readable by everyone so this would not be good. We can place the credentials for the smbmount in a hidden secured file in our home folder eg. /root/.smbereds echo username=user > .smbcreds ho password=pass >> .smbcreds chmod 600-smbereds ‘Then in the /etc/£stab file we substitute the usemame and password for credentials=/root/.smbereds. ‘Therefore the whole line in the fstab would be //sexver/share /mountdir smbfs ip=ipaddress, credentials=/root/.smcreds,noauto 0 0 ‘The noauto option specifies that this mount point should not be automatically mounted at boot. The administrator will mount and umount this as is required. ‘The 0 0 at the end of the line specifies backup pass and fSck pass ‘The backup pass flag relates to backup methods and generally you won't be backing up remote mounts in the service console, ‘The fick pass flag is a kind of dirty flag, Ifa file system were un-cleanly closed, then fick would check that flag on next boot and fix any errors found. Again, in the ESX service console, it's likely your mount points will be remote and will employ their own file system checks. ‘Therefore, your custom entries in /ete/fstab will typically terminate with the text noauto 0.0 vmfs_ftp Binary equivalent of ftp client, both vmfs_ftp and ftp are stored in /usz/bin lynx ‘This is a text mode web browser. Not that crazy, but a useful tool! ‘Try it with this web page if you need ‘command line help! wget ‘An http file get utility. For example, if you were at the command line and you needed the fix script for \Mtools install for SuSe Linux you could use: \wgelt http/woody lini org/vmeonffix.sh Shell ~/.bashre ‘A hidden file that extends shell script for the BASH shell. This is a hidden file that is found in each users home directory. ~/.bash_history‘Another hidden file which stores the previously entered commands by the user, i.e. a command history. ~/.bash_logout ‘A hidden script file that executes when a user logs out. The default content of this file is # ~/.bash_logout clear /etc/bashrc ‘System-wide bash shell settings. ‘set mode If you are a dedicated fan of the vi text editing tool then you bring it's functionality to the command line interpreter with this environment setting. history This command lists the commands you have previously entered along with a numeric index ID for each one. To re-use one of your previous commands, just enter an exclamation mark followed by the numeric ID of the ‘command you wish to re-use. For example, here we are using the history to view the commands and then re- using one by its numeric ID. @esxl 1 vdf -h 2 1s -k eat /ete/hosts 1i]$ history ali Feb 4 2005 fruit ali Nov 23 2004 lilo.conf.backup ali Feb 4 2005 scores ‘The history command is dependent upon the ~/.bash_history file being present and populated with data. ‘A great variation on this is just to use CTRL-R at the command line. This brings up a searchable command history which is very powerfil...try it out! clear ‘The clear command clears the terminal of existing output and returns the cursor to the first line. ‘This is the equivalent of the CLS command found in MS-DOS and the Windows command prompt. A greatshortcut way of doing a clear is CTRL-L, what is very cool, is that if you are in the middle of typing a command you can do a CTRL-L and the screen is cleared but your command line is still maintaine sum Prints the 16-bit checksum and size of the specified file. md5sum Prints the MDS checksum for the file supplied as a command parameter. VMware publish md5 ‘checksums for all their downloads on the www.vmware.com website to allow the end user to confirm that the file had not be corrupted in download. cal Display calendar for current month or set of months. The following command displays 3 months, current month and the month before and after. # cal -3 March 2006 April 2006 May 2006 Su Mo Tu We Th Fr Sa Su Mo Tu We Sa Su Mo Tu We Th Fr Sa 1234 1 12345 6 5 6 7 8 91011 2 3 45 67 8 7 8 910 213 2131415161718 9101112131415 1415 1617 18 19 20 19 20 21 22 23 24 25 16171819 20 21 22 21 22 23 24 25 26 27 26 27 28 29 30 31 23 24 25 26 27 28 29 28 29 30 31 ‘Surprisingly useful! /etc/vmware/vm-list This is an auto-generated text file listing the VMs on the ESX Server. This file should not be manually edited and should match up with output of the command vmware-cmd -1 ‘When you register a VM, itis automatically added to this file. The order in which VMs appear listed in the ‘MUI is dependent upon the order in which the VMs are listed in this file. Process Management PID PID stands for Process ID. Every running process has 2 process ID that is valid for the length of process execution. Use the ps command to view the service console processes and their associated PIDs.PPID PPID stands for Parent Process ID, which is the PID of the process that launched that process. ps ‘Show running processes in the service console. ‘The -f switch is useful as the “w" indicates wide format, so we can see the full directory path to the vmx file. ‘Another good option is the H option to show the process hierarchy in a similar way to pst ree. Ps -et ‘which might keep Solaris people happy as we don't have the pt ree utility in Linux. pstree -h ‘Show process tree and ancestors. root@esxl rootl# pstree nit ‘crond -gpm -httpa ~keventd -khuba -4*[kjournald] -klogd -5*[mingetty] si_eh_0 -snmpd -sshd- -syslogd -vmfs_flush -vmklogger -vnkst: -vmware-ccagent---vmware-ceage -5* [vmware-vmx-+-vmware-mks |-2* (vmware-vmx] ] . vmware-vmx] *-xinetd 3* [httpd] ‘You can display this process hierarchy with process ID numbers (PID) using the -p switch, To specify thatthe utility lists the processes with their command line arguments the -a switch should be used # pstree -ap renice Change process priority. If there were many VMs running concurrently, it may be necessary to temporarily raise the priority of the MUI in order to complete administrative tasks ps -eaf |grep vmware renice -p enice -p ‘To reset the PID of these processes back to their defaults, use renice again to set the priority to zero. pidof Finds the PID (process ID) of a named process. # pidof vmware-authd bg This is used to place a process in the background. For example, if we started a process at the command line that was time consuming and we wanted to work on something else, we could suspend that process with the CTRL-Z key sequence, and then place that suspended process in the background using this command. For example: a bg jobs Now the job will be running in the background, If you want to start a process in the background just add a "&" to the end of the command. fg ‘The foreground command, used to bring 2 background job back to the foreground for processing. For example S$ sleep 900 & jobs Lkdjsdlk S fg 1nohup If you run a process in the background and then log out, your process will be terminated. However, if you use the nchup command as a prefix to launching your command in the background, then your process will continue to execute in the background until it terminates, 1 still need to launch the process in the background when using the nohup command, i.e. after your ‘command you need an "&" character. For example: We can configure processes to run in the background by adding a “&” suffix when launching the program. # sleep 10 & If we know a process will take a while and we need the interactive command prompt back, then it's casier to launch the process this way, rather than start il, CTRL-Z. and using bg to place it in the background. jobs List the process jobs running in the background. SAN & Disk Management Disk.MaxLUN This is a VMkernel parameter setting. By default this value is set to 8, which means the VMkernel scans LUNs 0 to 7 on start up. So, if we what to scan up to LUN number N we must set Disk.MaxLUN to N+1, Ifyou change this setting away from the \VMkernel default, then the following file is created (or modified ifit exists already), /etc/vmware/vmkconfig ‘The safest way to update this parameter is by using the MUI in the Options tab, Advanced Settings. If you ‘wish to inspeet this value in the command line you can # cat /ete/vmware/config/Disk/MaxLUN Disk.MaskLUNs ‘This parameter controls LUN visibility, again this Is in the /etc/vnware/vmkconfig file. LUN masking Is only supported on fibre channel HBAs. This overrides the Disk.MaxLUN setting. vmhba0:0:4,6-255 would scan 0,1, 5 i.e, skip 4 and skip 6 through 255vmhba0:0: 255 would scan 0,1,2,5,6,7.8 ic. skip 3 & 4 and skip 9 through 255 Ifyou have multiple paths to LUNs you wish to mas ali available paths to those LUNs. you will need to supply a mask that masks LUNs on /etc/vmware/vmkconfig ‘This is a text file which stores VMkernel configuration, like 2 Windows INI file or Windows registry key. ‘The important fact to note is that this file does not exist until you make an edit away from the default setting, If you want to remove a setting from this file, either remove the offending line from this text file by manually editing it, or you can use the MUI (Options Tab, Advanced Settings) and enter a value of " (i.e, two quotation marks). Unfortunately as of ESX 2.5 just deleting the existing value in the MUI will not work. vmkfstools -s Used to re-scan for new LUNs on specified host bus adapter. Supply this command with the vmhba name of the HBA you wish to rescan, for example vmkfstools -s vmhba0 However, this has been known to cause problems in the past, hence the development of a script called cos~ xescan. sh to help. wwn SAN devices are identified by @ world wide name, @ unique 64-bit address. Remember we can use the perl script nwon.p1 to determine quickly what the WWN is for the installed FC hba LUN 'A LUN identifies individual units of storage behind a SCSI 1D. A LUN could be a single disk, a RAIDI volume, @ RAIDS volume or a logical partition of a RAID volume. For administrators of ESX Server, @ LUN is simply a unit of storage that is presented from SAN. Zoning Zoning is either hard (switch port) or soft (WWN controlled). LUN Masking LUN Masking Is a disk array feature that controls which LUNs are presented to which WWNs. The term ‘selective presentation Is also used, particularly in HP kit. /proc/scsi/driver/number‘The WWN for an adapter would be found in this file, k.SupportSparseLUN ‘The setting Disk. suppoztsparseLUN should =1 as LUNs may be discontiguous. For example if there are disk volumes at LUNs 0,1,2 and 6 then we want to be sure that after LUN 2, the VMkernel storage driver does not stop scanning. We want the LUN scanning to reach the last LUN specified in Di skMaxtim parameter, regardless of whether the visible LUN numbers are contiguous or not. Disk.RetryUnitAttention=1 ‘This setting tells ESX server to retry SCSI commands as vendor specific status codes may have been received and ESX may think the volume Is present but not accessible when in fact Its just a message that cache has been upgraded, vmkmultipath Allows SAN multipath maintenance from the command line, instead of MUI, Options, Storage Management. vmkmultipath -q Query multipath vmkmultipath -s -p policy sets path policy vmkmultipath -s -r path setsactive path vmkmultipath -S save configuration Performance & Monitoring free ‘A simple Linux utility to display available free memory in the service console. sxl vmimages] # Frustratingly, this doesn't have a -h switch for human readable as the d£ tool does, so we need to specify -k, -m or -g for kilobytes, megabytes and gigabytes respectively, esxtop ‘This is the VMware version of top and provides CPU, memory and disk configuration real-timeInformation just like the service console utility cop does, but this time we only see the data relating to the VMkernel so we see worlds instead of processes as resource consumers. 6:38pm up 2 days, 4:59, 17 worlds, load average: 0.00, 0.00, 0.00, 0.00 PCPU: 1.26%, 0.00% : 0.63% used total LCPU: 1.26%, 0.00%, 0 0 MEM: 3931136 managed(KB), 3668992 free (KB) : 6.67% used total SWAP: 4127744 av(KB), 0 used(KB), 4086468 (KB) : 0.00 MBr/s, 0.00 MBw/s: 0%, 0.008 vmhba0:0:0: 0.00 r/s, 0.98 w/s, 0.00 MBr/s, 0.00 MBw/s 0.00 r/s, 0.00 w/s, 0.00 MBr/s, 0.00 MBw/s vihbal vmhbal : 0.00 r/s, 0 MBr/s, 0.00 MBw/s vmhbal:0:1: 0.00 r/s, 0.00 MBw/s vihbal 0.00 r/s, 0.00 MBw/s NIC vmnic2: 0.00 pTx/s, 0.00 0.00 MbRx/s vmnicl: 0.00 pTx/s, 0.98 0.01 MbRx/s NIC vmnicO: 0.00 pTx/s, 0.00 0.00 MbRx/s 00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 top ‘Shows the running processes in the service console and lists the top consumers of CPU time. So if ‘someone is running something silly in the service console like @ DOOM network server! 6:4épm up 2 days, 5:06, 1 usex, load average: 0.00, 0.00, 0.00 42 processes: 41 3, 0 zombie, 0 stopped CPU states: 0.6% user, 0.4% system, 0.0% nice, 9.0% idle Men: 191240K av, 5928K free, 12280K buss ‘Swal Kav, 3 36980K cached ksoftirad_cPud kreclaind ush3 ° 0 0.0 0.0 oO: 9 0 o0 3 oO 0.0 0.0 0 9 0 0 o oo 0.0 0 ° 9 0 oo o oo 0.0 0.0 0 9 0 0 o oo 0.0 0.0 0 9 90 0 o 0 0.0 0.0 0 39 0 0 G 0 0.0 0.0 0 9 0 o0 3 Oo 0.0 0.0 0 9 0 472 460 o.0 0 ° 1198 roo: 9 0 23252 22M 0.012.1 7:17 v gent ‘The sample output shown above is static, but the actual output of the tool is continually changing as the processes are running. jostat vmstat Don't get confused by this command, it is a Linux command, not 2 VMware command. This is meant to view Linux processes, memory and paging. We generally use the vnstat command with 2 numeric parameters, the first parameter is how frequently the tool should run (specified in seconds). The second parameter specifies how many times the tool should run e.g. vmstat 2 3 ‘would run the tool every 2 -onds for 3 times and then exit, producing an output similar to the following: procs memory swap ° cpu x b w swpd free cache siso bi bo in cs us sy il 0 0 0 0 13132 64924 0 0 0 28 20 4 4 0 0 0 0 13132 64924 0 0 0 0 168 517 0 0 100 0 0 0 0 13132 64924 0 0 0 0 167 500 0 2 98 Under the swap heading in the output are the column headers "si" and "so” which correspond to swapped-in and swapped-out. Under the procs heading, the rb w column headers correspond to r= process is in run queue bb = process is blocked for resources /O Ww= process is swapped ‘The general rule is that ifr is consistently greater than the number of physical processors in the box, then the system will be slow. However, given that the ESX Server service console can only use physical CPUO, the service console rule should be ifr is consistently greater than 1, the service console will be slow, directly impacting your ability to manage the ESX server. Poor performance could manifest itself as poor MUI or remote console performance. Under the epu heading, the us sy id headers correspond to us = cpu user time sy = system time id = idle time ‘There is something (o be aware of in the Linux service console about swap. If the service console runs out of ‘swap, then the survival instincts of the Linux kemel kick in! Linux will kill off other processes at random tokeep itself alive, watch for this should your MUI go down, don't just re-start it, check why it stopped by ‘checking service console RAM using the £ree command and the vmstat command. man cpu Great info on using the /proc/vnwaze file system to change share allocation and processor affinity. /proc/vmware ‘This is not @ “real” directory, but is an in-memory volatile view of whet is happening inside the \Mkernel. By examining the contents of the files in the various directories within /proc/vmwere, we can gain valuable information real-time information. A great start is to look in the file neninzo /proc/vmware/memin In a subdirectory called /vm there are subdirectories for each vm labelled by number How do you find out ‘what number corresponds to what VM? Well, you can either just look in the "Status Monitor" tab in the MUI, fr go to the command line and run B2V's listworlds pl script or search each vmware . Log file for each VM and locate it's worldid: grep -i worldid= ~ali/vmware/1SAserver/vmmare. log ‘Some other examples of great info that can be extracted are outlined in the following table: /proc/vmware/vm/139/cpu /proc/vmmare/vm/139/cpu /proc/vmware/vm/139/cpu /proc/vmware/vm/138/cpu hyperthreading /proc/vmware/sched/epu vmkusage ‘This VMware-supplied tool generates web page usage reports on the ESX server as an extension to the MUL. This was not automatically installed In previous versions of ESX. wmkusagect1 install to install the utilisation web pages & setup a cron job vmkusage -graph to generate graph images. ESX 2.5.0 vmkusage-v2.5.0v2 ‘Sometimes this tool loses track of which VMs it should be charting. When this happens try using wmkusa ge -rescan IC that fails, then the more aggressive commandvmkusage -regroove ‘can be used to to wipe the charting data database and start logging stats again. Ifyou are doing specific troubleshooting, the vmkusage tool has a hyper switch which enables sampling data at a much higher frequency, but logging is only recorded every 1 minute. It is recommended you only use hyper on non-production systems and even then only for specific troubleshooting. wmkusage - hyper ‘There is another feature of vmkusage which allows generating of text performance reports at the command line. vmkusage - report -reportfile /home/ali/vmkreport ‘There is a tool called vmktree which many customers like to use with this tool, it can be found at httov/tihide.org/larstrvmktree/ hstatus ‘This is a web page like the MUI accessible via Inpufesxserverhstatus Not sure if'we need to be logged in for this to work, but vmkusage does appear to be required. We get loads of output on this page, similar to rmning a number of command line tools. This is part of what looks like the legacy web interface to ESX server, i.e. it doesn’t look as cool as the MUI of ESX 2.x. This legacy web output is not supported in the current release and could contain erroneous information. crontab Lists the cron jobs scheduled for the user when used with the -I (list) switch. For Windows users, think of cron as being like "Scheduled Tasks". ‘When customers install vmkusage they see a message stating that a cron job has been added, however crontab will not list the new job as it is not added under the context of the root user id. Perl Scripts vmsnap.pl Backup script to backup a virtual disk whilst the virtual machine is still running. This is achieved by the script creating 2 REDO file and then a REDO of the REDO file, thereby reducing the changing data and allowing a straight copy of the non-changing file. We supply the script the ‘mx file as a parameter to this script which will be read to find out where the virtualisk is. If we use this script with the -I switch we are stating that the resulting snapshot should be stored on the local server. The default path will be /vmimages/backup? If we choose to store the archive on another server, then the snapshot still occurs locally, but once complete, the archive is copied automatically using scp to the target archive server. Remember the backups produced are crash consistent, This means when you restore a snapshot image, the (OS will start a file system that was not cleanly shut down, and is therefore consistent with an OS that has just crashed. In the following example, the -I switch has been used to specify the backup only occurs locally. [root@esxl root]# vmsnap.pl -c /home/vmware/w2k/w2k.vmx ymsnap: VM config file is '/home/vmware/w2k/w2k.vmx vmsnap: This VM will only be backed up locally. ymsnap: Disks found that are in use: Vmsnap: Disk found: scsi0:1 (VMFS-For-All-Servers:w2k.vmdk) Wmsnap: 1: scsi0:1 ymsnap: Creating the RedoLogs for cfg : /home/vmware/w2k/w2k.vmx ymsnap: scsi0:1:Adding REDO ymsnap: scsi BACKUP STARTING NOW... local dir : /vmimages/localbackup has 5432340k space /vmfs/VMES-For-All-Servers/w2k.vmdk size : 2097152 k 2) diskname = VMFS-For-All-Servers:w2k.vmdk 2) disknameprefix = VMFS-For-All-Servers:w2k.vmdk 3) disknameprefix = VMFS-For-All-Servers:w2k 4) disknameprefix = VMFS-For-All-Servers:w2k disknameprefix = VMFS-For-All-Servers:w2k Exporting disk VMFS-For-All-Servers:w2k.vmdk: Export: 100% done. vmsnap: sesi0:1: Adding stacked REDO.REDO wmsnap: scsi0:1: Committing REDO Please wait, committing disks... ymsnap: scsi0:1: Committing REDO.REDO wmsnap: Backing up the cfg : /home/vmware/w2k/w2k.vmx [root@esx1 root] # LY some users have ran nto problems with vmsnap .p1 reporting that their virtual disk is already in REDO. mode and cannot be snapshot backed up. Be sure to check that previous snapshot backups ran successfully and there was sufficient disk space. vmres.pl‘This script is the restore tool for vusnap.p wwpn.pl ‘This tool is used to determine the world wide name (WWN) for a hba (fibre channel adapter). The same Information can be found in the MUI under Storage Management, but this Is nice and quick if you've already got a command line open. root@esxl root]# wwpn.pl \Vmhbal: 210000e08b17b3£6 (Qlogic) 6: If you run the script with the -v switch for verbose output, you also get the WWN information for the storage processors on SAN as well as the WWN for the ESX server hba. root@esxl root]# wwpn.pl -v 02 Copyright VMware 2003 Ww port names and VMHBA information for fibre channel adapters vmhba here are the corresponding Qlogic and Emulex WW Port Names WWEN PCT (decimal) 210000¢08b17b3£6 (Qlogic) 6 /proc/scsi/ala2300/0 vmhbal:0: 500508b30090ec31 scsi-qla0-port- 0=500508b30090ec30 : 500508b30090ec31; vmhbal:1: 500508b30090ec39 scsi-qla0-port- =500508b30090ec30:500508b30090ec39; The utility is called wwpn because it is listing the worldwide port name for your fibre channel hba, What can. be confusing is that there is a WWN for the node, pbind.p! Create or delete HBA persistent bindings. vmware-mount.p! ‘This is used to mount a virtual disk in the service console while the virtual machine is not running. For ‘example, if a configuration file in the guest OS virtual disk is preventing 2 virtual machine from booting correctly, you could use this utility to mount that virtual disk in another working VM. vmAddRedo.pl ‘This script adds a redo file to an existing VM. wmAddRedo [-c config file] [-g] [-h] [-m] Specify a VM configuration file to use for List all available VM's for backup Help m Generate the man page r this programIn the following example, we add a redo file to the VM called SUNone. As shown, the script parses the configuration file and then creates a redo file of the same name as the virtual disk but with a file extension \REDO. This can be performed against a running VM. [root@esxi root]# vmAddRedo.pl home/ali /vmware/SUNone/SUNone. vmx home/ali/vmware/s VvmAddRedo: VM config file is * ne/SUNone.. vmx vmAddRedo: Disks found that are in use vmAddRedo: Disk found: scsi0:0 (VMFS2-VvoL. vmaddRedc 1: sesi0:0 ymAddRedo: Creating the RedoLogs for cfg /nome/ali/vmware/SUNone/SUNone.vmx vmaddRedc scsi0:0:Adding REDO vmAddRedo: diskname :/vmfs/VMFS2-VOL1/sUNOne. vmdk :SUNOne. vmdk) vmCommit.pI This script commits a redo file to the virtual disk file. wmCommit [-c config file] [-g] [-h] [-m] -c config file Specify a VM configuration file to use for vmCommit -g List all available VM's for backup -h Help -m Generate the man page for this program vmware-config-tools.pl ‘This script is used to reconfigure the VMware Tools installed inside a Linux VM. cos-rescan.sh This script can be used instead of vk vmware-config-mui.pI ‘This script will regenerate the MUI SSL certificates. If you ever change an ESX Server hostname, you'll need to run this script. See the section on SSL for more details. /usr/sbin/vmkstatus ‘This Is a perl script which runs every x seconds on tty. ‘This Perl script runs on virtual terminal 1 (tty1) of the ESX server and provides the default console screen. ‘This script is bound to tty1 in the initialisation table /etc/inittab. This script can be run at anytime