VMware and Cisco Performance: Fact vs. Fiction
When is a virtualization benchmark really a virtualization benchmark? When is a virtualization benchmark a cloud benchmark?
Cisco has recently claimed that they have achieved “record virtualization performance” on an “industry standard cloud benchmark” with VMmark. Let’s look at the facts:
Fact: As a benchmark, VMmark is ultimately limited by CPU performance and has limited scope for comprehensively measuring overall virtualization. IBM studies consistently show that CPU utilization is often not the performance constraint for x86 virtualization. Fact: VMmark is certainly not an “industry standard” benchmark as it was not developed through open collaboration by the industry in the manner that TPC or SPEC benchmarks are developed. Fact: The “world record” result Cisco claims on VMmark needed to use 4 Cisco UCS C460 hosts — not just 2 hosts that have been used in every other VMmark 2.0 published result — and used 160 cores and 320 threads. Fact: SPECvirt_sc2010 is THE industry-standard virtualization benchmark. SPECvirt_sc2010 measures the end-to-end performance of all system components including the hardware, virtualization platform, and the virtualized guest operating system and application software. IBM is currently #1 in this benchmark.(1) IBM has outstanding results on this virtualization benchmark, whatever virtualization technology is used. Where are the results here from Cisco on this one and on other SPEC and TPC true industry standard benchmarks ? Fact: VMmark is certainly not a “cloud” benchmark. Cloud complexity is awe-inspiring with its huge scope, variability, and integration. Many organizations such as SPEC are currently working on developing sophisticated cloud benchmarks and will need to focus on consumability metrics such as deployment, elasticity, resiliency, backup, and maintenance.
Calling it virtualization performance does not make it virtualization performance. Calling it cloud does not make it cloud.
(1)SPECvirt_sc2010 highwater result on the IBM x3850 X5 (8 Intel Xeon E7-4870, 2.4 GHz, 80 cores, 10 cores/chip, 2 threads/core) with 2TB memory of 7067@432 SPECvirt_sc2010@VMs running Red Hat Enterprise Linux 6.1 (KVM).
Source: http://www.spec.org Results current as of 9/9/11.
SPEC, SPECint, SPECfp, SPECjbb, SPECweb, SPECjAppServer, SPECjEnterprise, SPECjvm, SPECvirt, SPECompM, SPECompL, SPECsfs, SPECpower, SPEC MPI and SPECpower_ssj are trademarks of the Standard Performance Evaluation Corporation (SPEC).
TPC-C ,TPC-H, and TPC-E are trademarks of the Transaction Performance Processing Council (TPPC).
VMware, the VMware “boxes” logo and design, Virtual SMP and VMotion are registered trademarks or trademarks (the “Marks”) of VMware, Inc. in the United States and/or other jurisdictions.
The postings on this site solely reflect the personal views of the author and do not necessarily represent the views, positions, strategies or opinions of IBM or IBM management.