VMware, CPU VT Features and Asus Motherboards

I recently built a machine to run as a lab/training box for VMware ESXi at home, and have it running quite nicely with Virtual Center and a few VMs.  I ran into a bit of trouble before Christmas though when I tried to run up a 64-bit Ubuntu Server.  

The VM wouldn't power on due to an error with a mismatch in CPU capabilities and those required by the VM.  

"cpu on host is incompatible with cpu feature requirements of virtual machine. problem detected at cpuID level 0x80000001 register 'edx'."

It turned out to be the 'VT' (Virtualization Technology for x86 (VT-x), previously codenamed "Vanderpool", referred to in the BIOS setup as 'Vanderpool Technology') feature that ESX was complaining was missing.  I was totally confused though as both my motherboard (an Asus P5QL-EM) and CPU (an Intel E8400) support the VT feature and it was indeed enabled in the Motherboard BIOS.  

I found the very useful tool VMware CPU Host Info and ran it up against my Virtual Center server.  Sure enough, it showed that although my ESXi Hardware supported VT it was not actually enabled.  So I doubled checked the BIOS setting again, enabled (after shutting down all running VMs, of course).  

After establishing that I was running the latest version of the BIOS (0412) I decided to try and contact Asus, through their website, to log this as a support fault.  I received the following reply (on 19/12): 

Dear Clint Revell,
Thank you for contacting ASUS Customer Service.

My name is Lyn and it's my pleasure to help you with your problem.
The motherboard supports Vanderpool Technology.
And it seems that you have enabled it in BIOS.
But as I know, Intel Vanderpool Technology has special requirements for the OS and the softwares too. For detail, maybe you can try to contact Intel directly for some help.
http://mysearch.intel.com/corporate/default.aspx?culture=en_US&q=Vanderpool+Technology&category=all&adv=0&value=0
Welcome to refer Troubleshooting & FAQ for ASUS products in ASUS website:
http://support.asus.com/troubleshooting/troubleshooting.aspx?SLanguage=en-us
http://support.asus.com/faq/faq.aspx?SLanguage=en-us

If having any problems, please don't hesitate to let me know. Let's discuss this issue together.
Thank you for using ASUS products and enjoying ASUS services!

Lyn

ASUS Customer Service Center (Shanghai, China)

I'm sure you'll agree that the response was pretty much useless.  But it was a response, and a nice polite one at that.

In the New Year I decided to try again, with the same result, and checked again for a BIOS update.  What'dya know... Asus released the P5QL-EM BIOS version 0418 on 30/12/2008 with the following fixes:

1. Fix "Vanderpool Technology" item is invalid in setup.

So I installed the BIOS update, my 64-bit Ubuntu VM booted up straight away my VMware CPU Host Info tool of course shows 'VT Enabled'.  Hooray!  

So did Asus fix the issue after I contacted them?  Or did they just notice it themselves and remedy it as part of the ongoing improvement process.  I guess I'll never know, but I'm glad it's fixed none-the-less.

VMware ESX3i Lab Machine at Home

After I finished the VMware Infrastructure 3: Install & Configure v3.5 course I decided to put together a cheap(!) PC that I could use to mess around a little more with ESX, particularly before I sit the VCP exam.  So I looked around for hardware that could reasonably run a few Virtual Machines, and ESX, in as small a form-factor as possible.  For a custom PC that mean looking for a Micro-ATX motherboard and Case, a CPU with VT and 64bit features, at least 4GB RAM and a reasonable amount of disk space.

The motherboard I decided on (after much deliberation) was an Asus P5QL-EM.  I really wanted to get a P5Q-EM but noone in New Zealand had them in stock.  As it was the company I got all of the other components from (Playtech) couldn't get me a P5QL motherboard, so I had to order it through Acquire and go and pick it up from Morning Star out by the Airport (quite a way from my house).  The main difference between the P5Q and P5QL is the chipset (G43 as opposed to the G45) which means less total memory capacity (8GB instead of 16GB) and its also missing a few Media-related capabilities.  I didn't think I'd miss those that much, and I wanted it sooner rather than later so went for the cheaper board.  Other cool things about this motherboard:

• Supports the 45nm CPUs (lower power & heat - good for home)
• Onboard VGA, DVI and HDMI (capable of full 1080p output - might be nice for a future media center PC)
• eSATA connector for additional storage

I bought an Intel E8400 Core2 Duo 3GHz, which is a 45nm CPU.  A bit more expensive than the older CPUs but I like the low power idea.  Also the HDD I bought is low power rated and quiet - a Western Digital 1TB EACS.

The case - a Silverstone SG03B is pretty funky looking. The guy at Playtech said I would struggle to build it, as it is quite cramped, but I didn't have any problems.  It was quite a cool little project to put together.  The Micro ATX case sure is cute, and fits everything in nicely.  Silverstone have done a good design job to make sure you can get at everything, removable HDD rails etc.  The Corsair HX-620 power supply has modular power connectors - so that cuts down some of the clutter inside the case.  

I installed 4GB of G.Skill DDR2-800 RAM, so there's still room for another 4GB if I need it.  I still can't believe you can get 4GB of RAM for $100 now!

I'm pretty happy with the final result.  It's a pretty quick machine, but nice and compact and really quiet.

So after building all this I tried to install ESX 3.5 (update 3) from the downloaded CD (ISO image).  As expected it wouldn't install due to a hardware issue.  The first one was due to the fact ESX couldn't read from the (SATA) CD-ROM.  It brought up the (rather unhelpful) error message "No driver found, Unable to find any devices of the type needed for this installation type".  

What it really wanted to say was change the storage configuration in the BIOS to IDE mode, and type the following at the ESX installation boot prompt:

esx all-generic-ide irqpoll pci=nommconf

So having gotten past that the expected issue showed up - ESX doesn't like the onboard NIC.  So I went on a mission to try and find an HCL-listed network card.  I wanted a Gigabit Card, with e1000 driver support.  So it came down to finding an Intel Pro 1000GT or Intel Pro 1000PT.  I couldn't find anyone with an Intel Pro 1000 PT desktop NIC (non-low Profile) in stock, so I had to front up with $200 for a Server NIC.  A nice card, but a little pricey all the same.

So ESX installed fine, and I jumped on the web interface, downloaded and installed the Virtual Infrastructure Client and logged on.  No storage adapters, and no ability to add VMFS storage :-(  So while I could connect to a NAS using NFS or iSCSI I couldn't use my 1TB drive for storage of any Virtual Machines VMDKs or ISO images.  Pretty much useless.

So after a few tears, and feeling very ripped off, I decided to go for the VI in a box setup.  Not desirable at all, but potentially the only way to get ESX running on my new hardware: Install Ubuntu Server 64bit (host), VMware Server 2.0 64bit and ESX Server 3.5 update 3 (guest) running in a Virtual Machine.  I've been trying on and off for the past week to get that to work.  I've read so many Blog posts about getting ESX to work in a VM, most or all with the same hints/tips.  

But while they work fine for VMware Workstation 6.5, they just don't work for VMware Server 2.0 (for me anyway).  ESX installs fine, but sits eating 100% CPU, takes hours to finish booting and when it finally does it's so slow its unusable.  Here are some of the better links I've found:

The Xtravirt Whitepaper: http://knowledge.xtravirt.com/downloads/ESXonWKS65118166.pdf

The VMware Forums: 

http://communities.vmware.com/thread/86877?tstart=0&start=45

http://communities.vmware.com/thread/67254?tstart=0&start=15

http://communities.vmware.com/thread/164935

Running ESXi in VMware Server 2: http://eightfour.com.au/index.php?option=com_content&task=view&id=33&Itemid=1

What they basically all say is you need to ensure your VMX file has the following lines added:

monitor_control.restrict_backdoor = "TRUE"

monitor_control.vt32 = "TRUE"

ethernet0.virtualDev = "e1000"

So then I started following some of the threads about ESXi, and downloaded the ISO, and tried to run that in a VMware Server VM.  But I had the same problem.  

So today I thought I'd try to install ESXi instead of 3.5 on the PC as a host.  Because although I'd found lots of articles saying SATA isn't supported AND doesn't work for VMFS storage, I also found a couple that indicated it DID work.  

http://vmblog.com/archive/2007/10/03/hacking-your-vmware-esx-server-3i.aspx

http://www.run-virtual.com/?p=196

I followed the instructions in this VMware forum thread:

http://communities.vmware.com/thread/158209?tstart=0&start=60

And now I've got a fully functional ESX Server - finally - with almost 1TB of VMFS storage space for VMs!

In summary the steps are:

• Replace the oem.tgz file on the ESXi Installer CD with one that includes the SATA controller drivers (ICH10 in my case - from here).
• Install ESXi
• Boot from a Live Linux CD (I used Gentoo Minimal) and have a USB stick with oem.tgz on it connected
• Mount the ESXi partition (mount /dev/sda5 /mnt/gentoo)
• Mount the USB stick (mkdir /mnt/usb) and then (mount /dev/sdb /mnt/usb)
• Copy the oem.tgz file from the USB stick to the ESXi installation (cp /mnt/usb/oem.tgz /mnt/gentoo)

Thanks to all those in the VMware forums and other blogs that contributed to that info.  It worked like a charm for me!

So now I'm getting on with setting up a vCenter Server (Formerly VMware Virtual Center - VMware have gone all Checkpoint and started renaming products every few months! - the weirdest one: Old: VMFS, New: VMware vStorage VMFS! ... who is going to say that mouthful every time?).  

I also might have a play setting up my NAS with NFS and use that for Vmotion between a couple of ESX servers. Maybe one running in a VMware Workstation 6.5 VM...hmmm, wonder if that'll work ;-)

VMware Infrastructure 3: Install and Configure Day 4

I just finished the fourth and final day of the VMware Infrastructure 3 course. I think overall it was pretty worthwhile. Definitely picked up a few good tips and learnt more about some of the more detailed features/functions of the products - especially the newer stuff. And best of all - I just got a nice new VMware bag :-) Cool. It's quite a good one too (better than the Cisco one from Networkers this year!).

Now to study for the VCP exam! (And maybe build myself an ESX Server lab machine at home to help me revise)

Here are my notes on the topics we've been through today.

Resource Management

• VMotion network can be used as a secondary heartbeat network for HA
• The final thing the VMkernel does after a VMotion is a Reverse ARP, ensuring the switch knows the Virtual Machine's MAC address is accessible via the 2nd ESX Host
• The time between VMotion and scheduling instructions on the 2nd ESX Host's CPU is typically between 50 and 200ms.
• For VMotion to work there must be a virtual switch and port group with an identical name (case sensitive) on the 2nd ESX Host.
• As CPU affinity can cause incompatibility with VMotion only use in a lab/testing environment.

Resource Monitoring

• HEC = Hardware Execution Context. An HEC is typically a CPU core (today), but also a HT CPU.
• vmmemctl = Balloon Driver. Takes RAM from guest machines when physical RAM in host is scarce, causing Guests to Page/Swap more.
• By default the Balloon Driver will not expand beyond 65% of the VM configured memory.
• Use iometer to generate artificial disk load for testing purposes

Data and Availability Protection

• Service Console backups are in reality limited to a small amount of configuration settings, 3rd party software/agents, and logs in /var/log partition.
• ESX Host restore can be scripted using anaconda kick-start files, using a finish script to recreate Virtual Switch settings etc.
• VMware HA does not increase availability, it decreases downtime.
• VMware HA restarts virtual machines when physical machines fail.
• VMware HA network ports:
• Incoming - TCP/UDP 8042-8045
• Outgoing - TCP/UDP 2050-2250
• VMware HA best practices: http://kb.vmware.com/kb/1002080

VMware Infrastructure 3: Install and Configure Day 3

And at the end of Day 3 on the VMware Infrastructure 3 course...more notes...

Virtual Machines...

• Open Filer is an Open Source Storage Appliance (NAS), including iSCSI support, runs well in a VM, available as a Virtual Appliance.

VMware Converter

• VMware Converter standard is free. Enterprise costs $.
• VMware Converter standard is used one machine at a time. Enterprise can run unattended converting multiple machines.
• VMware Converter standard can be operated in 'Hot' and 'Remote' modes, Enterprise is required for 'Cold' operation (boot from CD).

Virtual Machine Management

• VirtualCenter can now expand volumes, similar to the command-line tool: vmkfstools

Guided Consolidation

• Limited value - potentially only for very small environments
• Only collects 24 hours of data, from 8 perfmon counters
• Confidence Level is really a progress bar with 100% representing 24 hours of data collected. High Confidence does not necessarily mean the data collected is representative of the environment's typical workload.
• VMware capacity planner is a much better tool, collects all statistics for 30 days, and works on both Windows and Linux

Access Control

• VirtualCenter authentication is Windows only (AD or Local Windows Accounts)
• ESX (Service Console) authentication is local Linux user accounts or PAM (e.g. LDAP back to AD)
• VirtualCenter authenticates to an ESX host as 'vpxuser' (with a random password)
• Permissions = User/Group + Role (+ Privileges). Permissions -> applied to -> Inventory Objects
• Web Access 'Generate Remote Console URL' provides access (directly to an ESX host) for users/administrators to manage a single VM.

Resource Management

• CPU or Memory shares are never used unless there is contention for CPU or Memory resources
• Shares are only relative within resource pool siblings - not between resource pools

VMware Infrastructure 3: Install and Configure Day 2

So today was day two of the VMware Infrastructure 3: Install and Configure course. It was a lot more interesting than Day 1, although still ran at a pretty slow pace. Below are some of my notes made during the course & labs:

Storage

• SAN Troubleshooting Guide available, recommended reading:
• http://www.vmware.com/pdf/vi3_san_design_deploy.pdf
• ESX can boot from an iSCSI SAN only when using a hardware initiator (iSCSI HBA).
• ESX does not currently support TCP/IP offload engine (TOE) NIC cards (although it does support TCP session offload).
• When using a software iSCSI initiator the session is established (TCP/3260) using the Service Console network interface (CHAP & SendTargets IP address discovery). Network Connectivity is required from the service console to the iSCSI SAN (could use a 2nd service console interface, just for this purpose).

VirtualCenter

• ODBC connection from Virtual Center to SQL 2005 DB needs SQL Native Client (not SQL Server ODBC driver)
• Virtual Center SQL Login needs db_owner on vCenter and MSDB databases

Important VMware communications - e.g for firewall security policy

• VirtualCenter to License Server: 27000, 27010
• Rebates and SDK Clients to VirtualCenter: 443, 80
• VirtualCenter to ESX Hosts: 902
• VI Client to VirtualCenter: 443
• VI Client to ESX Host Remote Console: 903

• By default VirtualCenter accumulates performance data on ESX Hosts and VMs for 1 year
• Commands to check the status of, or start the ESX host process (vmware-hostd):
• service mgmt-vmware status, service mgmt-vmware start

Virtual Machines

• As a guideline deploy single CPU VMs unless a multiple CPU VM is absolutely needed (e.g. multithreaded application requirement).
• As a guideline deploy multiple CPU VMs when there is at least a 2:1 ratio between virtual and physical CPU requirements (e.g. dual core VM, have quad core Host. Quad core VM have 8 core Host).
• There is no support in ESX for virtual USB devices (even though there is in VMware Workstation).
• www.digi.com have a USB to TCP/IP (5xUSB, 1xRJ45 - 10/100Mb Ethernet) converter - called AnywhereUSB. Software can be installed in the VM which provides access to the USB device over the VM's network interface.

VMware Infrastructure 3: Install and Configure Training

Today was day one on the 'VMware Infrastructure 3: Install and Configure' training course, which I'm doing mainly because that's the only way VMware will allow you to get the VMware Certified Professional (VCP) certification (you can't just take the exam - the course is required). As the course is pretty slow-paced I thought I'd blog some notes as I go along, which I can refer back to if needed, and they may come in handy for someone else out there too...

Certification & Exams

• Read the VCP on VI3 Exam Blueprint
  


• http://mylearn.vmware.com/lcms/mL_faq/1714/VCP3.5Blueprint.PDF
  


• Take the VCP on VI3 Mock Exam
  


• http://mylearn.vmware.com/quiz.cfm?item=639&ui=www
  


• Book the Exam through Pearson Vue (90 mins, 70% required):
  


• http://www.pearsonvue.com/vmware/

Intro

• VMware Authorized Consultants (VACs) can access a tool, VMware Capacity Planner, which helps plan virtualising a data centre.
  


• VCDX Certification intended to represent a similar 'level' of ability in VMware that the CCIE did 10 years ago. Intended to be very difficult to attain. Requires multi choice exam, hands-on labs and completion and presentation of a design to a board of examiners.
  


• The HCL is three documents, updated almost every week (Wednesday). System (Server), IO (NIC), Storage/SAN.

Installation

• swap Partition size = 2 x Service Console RAM (272MB) = 544MB (No 3rd party modules) = 2 x Service console RAM Maximum (800MB) = 1.6GB (Recommended for Production Use, permits the installation of additional 3rd party modules in the service console)


• /var/log = 500MB minimum, VMware recommends 2GB.

Labs

• Accessed the 'Lab' via VMware's 'Virtual Datacenter' - actually HP Proliant DL360G5 servers in one of two physical datacenters in San Francisco. (VMware have 700 physical hosts to run education around the world). Installed ESX 3.5 using HP iLO access, from a Citrix Metaframe XP connection to a VMware Classroom 'PC' (running the RES PowerFuse locked-down Windows shell).


• Commands for troubleshooting service console connectivity (wrong NIC selected as service console):


• esxcfg-nics -l (list NICs and PCI addresses in system)


• esxcfg-vswitch -l (list NICs and virtual switches)


• esxcfg-vswitch -U and esxcfg-vswitch -L (to link the correct NIC)

Networking

• ESX has three uses for networking:

1. Virtual Machine connectivity


2. Service Console connectivity


3. Kernel Connectivity (Vmotion + Storage (iSCSI, NAS))

• Load balancing VM networking connections across multiple physical NICs based on IP Hash requires Etherchannel/802.3AD Aggregation support in the physical network switching infrastructure.