Virtualization shoot-out: Citrix, Microsoft, Red Hat, and VMware

In data centers large and small, the move to server virtualization seems as unstoppable as the waves crashing on the beach in Hawaii’s Waimea Bay. But for almost as long as the virtualization tide has been rising, there was only one vendor that could offer the features, interoperability, and stability necessary to bring virtual servers out of the skunkworks and into daily production. That is no longer the case.

From the beginning, VMware has been the king of x86 server virtualization, hands down. VMware’s feature set, reputation, and pricing all reflect that fact. But where there used to be little competition, you’ll now find a select group of challengers that have brought a wealth of enterprise features to their virtualization solutions and begun to give VMware a run for its money.

In order to accurately gauge just how close this race has become, we invited Citrix, Microsoft, Red Hat, and VMware to the Advanced Network Computing Lab at the University of Hawaii, and we put their server virtualization solutions to the test. We compared Citrix XenServer, Microsoft Windows Server 2008 R2 Hyper-V, Red Hat Enterprise Virtualization, and VMware vSphere by virtually every measure, from ease of installation to hypervisor performance, and all of the management capabilities in between.

We tested each solution on the same hardware, with the same real-world network topology, running the same tests on the same virtual machines. We ran real-world and synthetic Linux and Windows performance benchmarks, and we performed subjective management and administration tests. We looked at host configuration, VM templating and cloning, updates and patching, snapshots and backups, and scripting options, and we examined advanced features such as load balancing and high availability.

The results showed that all four solutions combine very good hypervisor performance with rich sets of management tools. But the solutions are not all equal in either performance or management. Although VMware is no longer the only game in town, choosing an alternative certainly involves trade-offs.

VMware still has advanced capabilities that the others lack. VMware also offers a level of consistency and polish that the other solutions don’t yet match. The rough edges and quirks in Citrix, Microsoft, and Red Hat aren’t showstoppers, but they demonstrate that these alternatives all have hidden costs to go along with their (potentially) lower price tags.

Virtualization shoot-out: The test bed The fine folks at Dell were kind enough to lend us a bunch of high-end gear to run all of our tests. We requested blade servers for a variety of reasons. Primarily, we wanted the ease of setup and configuration offered by the blade chassis, which consolidates the power, network, and remote management into a single unit. We chose the two-socket blades for our test servers, as these are more representative of production hypervisor configurations than other CPU densities.

We were equipped with a Dell PowerEdge M1000E chassis with two Dell PowerConnect M8024 10G switch modules and a PowerConnect M6220 gigabit switch module. The storage tasks were easily handled by a Dell EqualLogic PS6010XV 10G SAN array, and we used four Dell PowerEdge M710HD blades to run the hypervisors. Each M710HD was equipped with two Intel Westmere 5645 CPUs running six physical cores at 2.40GHz. Those were accompanied by 96GB of DDR3 RAM, dual-port Intel X-520 10G Ethernet mezzanine adapters, and built-in dual-port gigabit NICs. Each server also had Dell’s redundant SD-based flash devices for embedded installations and a pair of 72GB SAS drives in a RAID1 configuration for hypervisors that required traditional installation.

For backline duties, we used two Dell PowerEdge M610 Intel Nehalem-based blades. These blades were not part of the actual test, but were used to provide supporting services such as Microsoft Active Directory, DNS, and DHCP. Suffice it to say, we were very well outfitted on the hardware front.

Virtualization shoot-out: World’s fastest hypervisor The test plan was straightforward: Take a look at Windows and Linux server performance on the physical hardware, then on an otherwise quiescent hypervisor, as well as several more times on a hypervisor under increasing load levels. Metrics included CPU, RAM, network, and storage I/O performance, time and interruption (if any) during VM migrations, speed and agility in VM template creation and deployment, and overall handling of a few disaster scenarios, such as the abrupt loss of a host and failover to an alternate site.

The benchmarks themselves were based on synthetic and real-world tests. They provide a general picture of hypervisor performance, but as with many facets of virtualization, there’s no good way to accurately forecast how any workload will perform under any virtualization solution apart from running the actual workload.

The Linux tests were drawn from my standard suite of homegrown tests. They are based on common tools and scenarios, and they’re measured by elapsed time to complete. These included converting a 150MB WAV file to MP3 using the LAME encoder on Linux, as well as using bzip2 and gzip to compress and decompress large files. These are single-threaded tests that are run in series, but with increasing concurrency, allowing performance to be measured with two, four, six, eight, and twelve concurrent test passes running. By running these tests on a virtual machine with four vCPUs (virtual CPUs), we were able to measure how well the hypervisor handled increasing workloads on the VM in terms of CPU, RAM, and I/O performance, as all files were read from and written to shared storage.

The Windows tests were run with SiSoftware’s Sandra. We chose to focus on a few specific benchmarks, primarily based on CPU and RAM performance, but also including AES cryptography, which plays a significant part in many production workloads.

Again, all tests were conducted on the same physical hardware, with the same EqualLogic PS6010XV iSCSI array for storage, and on identical virtual machines built under each solution. All the Windows tests were run on Windows Server 2008 R2, and all the Linux tests were run on Red Hat Enterprise Linux 6 — with the exception of Microsoft Hyper-V. Because Hyper-V does not support Red Hat Enterprise Linux 6, we used RHEL 5.5, which may have had a minor impact on Hyper-V’s Linux test results.

The performance test results show the four hypervisors to be closely matched, with no big winners or losers. The main differences emerged in the loaded hypervisor tests, where XenServer’s Windows performance and Hyper-V’s Linux performance both suffered. Overall, VMware vSphere and Microsoft Hyper-V turned in the best Windows results [see table], while vSphere, Red Hat Enterprise Virtualization, and Citrix XenServer all posted solid Linux numbers [see table]. The crypto bandwidth tests, where XenServer and vSphere proved three times faster than Hyper-V and RHEV, showed the advantages of supporting the Intel Westmere CPU’s AES-NI instructions. Charts of a few of these test results are displayed below. 

Virtualization shoot-out: Understanding the spread Although VMware still leads the pack in features, not all of them carry the same weight when placed in any given corporate environment. Three features that will matter to environments of any size are live VM migrations, high availability, and load balancing.

Live migrations are the ability to move running virtual machines from host to host without rebooting. High availability allows a solution to determine when a physical host is down and automatically restart the virtual machines that were running on that host elsewhere in the farm. Automated workload balancing levels the VM load across multiple farm servers based on thresholds set by administrators. These are key features that once were available only from VMware, but are now present in each solution.

Following those features are slightly less important ones such as thin provisioning, VM snapshotting and templating, automated VM backups, live storage migrations, and advanced memory management features, including page sharing, memory compression, and memory ballooning.

VMware is still the only solution that can handle live storage migrations, and only VMware vSphere and Red Hat Enterprise Virtualization can claim to offer page sharing and memory compression. However, the other features are available across the board. Thin provisioning and templating are features that make deploying large numbers of VMs trivial, and at the same time reduce the impact on storage. Advanced memory management capabilities optimize the allocation of physical memory to virtual machines.

These tools are extremely important in infrastructures running a large number of VMs per host, and a large number of VMs overall, but smaller environments may not need them straight away. While definitely important, these features do not have as direct an impact on end-users as live migrations, high availability, and load balancing, especially considering the number of smaller organizations that are planning on virtualizing their entire server farm and will experience less churn than larger infrastructures.

Further down the feature trail, we get into more rarified air — and items like API integration, which can provide massive benefits to larger implementations. All four solutions offer some form of CLI or scripted management, ranging from VMware’s mature API integration to Hyper-V’s use of PowerShell to allow for scripting administrative tasks.

There’s also something to be said for guest OS support. Only VMware goes as far as explicitly supporting Mac OS X Server, FreeBSD, NetWare, and Solaris. You can even run OS/2 Warp if you like. The rest stop at a few Linux distributions and Windows platforms.

The long and the short of it is that at this point in time, all of these vendors can provide enough features to consider their use in a production capacity in a wide variety of deployment sizes and types. However, VMware still has the most mature and feature-rich offering among them. You’ll find a features comparison table on the next page; the article continues on the following page. 

Virtualization platform licensing The licensing structures of VMware vSphere and Microsoft Hyper-V are definitely more complex than those of either Red Hat Enterprise Virtualization or Citrix XenServer. VMware offers several levels of vSphere, each with more features than the last, and all priced per physical socket. Microsoft offers Hyper-V as part of Windows Server 2008 R2, with an Enterprise license allowing for four virtual servers running the same OS on a physical server, and a Datacenter license allowing for unlimited virtual machines per physical server.

Interestingly, many shops are buying Microsoft’s Datacenter licenses and assigning them to physical servers running VMware vSphere. Those licenses allow for unlimited Windows Server 2008 R2 VMs on that host, even if it’s not running Hyper-V.

Citrix XenServer is priced per server, regardless of server capacity. Like VMware, Citrix offers a choice of several tiers. Red Hat Enterprise Virtualization is the simplest (and cheapest), with flat per-year subscription pricing per physical server based on 9-to-5 or 24/7 support starting at $499 per server per year.

Virtualization now, more than ever The major excuse for delaying a virtualization project in the past was the price of VMware balanced against the lack of significant features such as live migrations, high availability, load balancing, and even guest OS support in competing products. That is no longer valid, as each solution capably demonstrated these features. Larger infrastructures may still view virtualization as VMware and VMware only, but the smaller and midrange shops that can live without VMware’s advanced features suddenly have a plethora of options. They can still bring the extreme benefits of virtualization into the data center and not bust the budget.

Among the three challengers, Microsoft Hyper-V comes closest to VMware vSphere in overall management functionality. However, whereas VMware, Red Hat, and Citrix combine virtualization host and VM management in a single management server, Microsoft spreads the functions across multiple System Center tools. Hyper-V’s advanced capabilities come at the cost of additional overhead, configuration, and complexity for administrators.

Citrix XenServer combines great Linux performance, rapid deployment, and the major advanced features, though high availability and load balancing require supporting players and additional configuration. A significant drawback is that all VM management operations are serialized. It takes significantly longer to perform any action (such as power up or power down) on multiple virtual machines in XenServer than in any of the other solutions — a limitation that impacts both manageability and scalability.

Red Hat Enterprise Virtualization has all of the essential management capabilities, but they’re nowhere near as complex to implement as the tools for Hyper-V. RHEV also has a few rough spots, including odd gotchas in host maintenance mode and high availability, but it boasts a quick install, solid Linux and Windows performance, a good CLI, and advanced memory management features such as page sharing and compression. It comes closest to VMware in having all the ingredients to support a scalable environment. 

It may come as little surprise that VMware vSphere still leads the pack by a handy margin, but the gap is closing fast. If there’s one obvious result of this test, it’s that there’s never been a better time to shop for a virtualization solution.

Read the reviews:

• Virtualization shoot-out: Citrix, Microsoft, Red Hat, and VMware
• Virtualization shoot-out: Citrix XenServer
• Virtualization shoot-out: Microsoft Windows Server 2008 R2 Hyper-V
• Virtualization shoot-out: Red Hat Enterprise Virtualization
• Virtualization shoot-out: VMware vSphere

This article, “Virtualization shoot-out: Citrix, Microsoft, Red Hat, and VMware,” was originally published at InfoWorld.com. Follow the latest developments in virtualization and cloud computing at InfoWorld.com. For the latest business technology news, follow InfoWorld.com on Twitter.