Elisabeth Stahl on Benchmarking and IT Optimization

VMware and Cisco Performance: Fact vs. Fiction

with 3 comments

When is a virtualization benchmark really a virtualization benchmark? When is a virtualization benchmark a cloud benchmark?

Cisco has recently claimed that they have achieved “record virtualization performance” on an “industry standard cloud benchmark” with VMmark. Let’s look at the facts:

  • Fact: As a benchmark, VMmark is ultimately limited by CPU performance and has limited scope for comprehensively measuring overall virtualization. IBM studies consistently show that CPU utilization is often not the performance constraint for x86 virtualization.
  • Fact: VMmark is certainly not an “industry standard” benchmark as it was not developed through open collaboration by the industry in the manner that TPC or SPEC benchmarks are developed.
  • Fact: The “world record” result Cisco claims on VMmark needed to use 4 Cisco UCS C460 hosts — not just 2 hosts that have been used in every other VMmark 2.0 published result — and used 160 cores and 320 threads.
  • Fact: SPECvirt_sc2010 is THE industry-standard virtualization benchmark. SPECvirt_sc2010 measures the end-to-end performance of all system components including the hardware, virtualization platform, and the virtualized guest operating system and application software. IBM is currently #1 in this benchmark.(1) IBM has outstanding results on this virtualization benchmark, whatever virtualization technology is used. Where are the results here from Cisco on this one and on other SPEC and TPC true industry standard benchmarks ?
  • Fact: VMmark is certainly not a “cloud” benchmark. Cloud complexity is awe-inspiring with its huge scope, variability, and integration. Many organizations such as SPEC are currently working on developing sophisticated cloud benchmarks and will need to focus on consumability metrics such as deployment, elasticity, resiliency, backup, and maintenance.
  • Calling it virtualization performance does not make it virtualization performance. Calling it cloud does not make it cloud.


    (1)SPECvirt_sc2010 highwater result on the IBM x3850 X5 (8 Intel Xeon E7-4870, 2.4 GHz, 80 cores, 10 cores/chip, 2 threads/core) with 2TB memory of 7067@432 SPECvirt_sc2010@VMs running Red Hat Enterprise Linux 6.1 (KVM).
    Source: Results current as of 9/9/11.

    SPEC, SPECint, SPECfp, SPECjbb, SPECweb, SPECjAppServer, SPECjEnterprise, SPECjvm, SPECvirt, SPECompM, SPECompL, SPECsfs, SPECpower, SPEC MPI and SPECpower_ssj are trademarks of the Standard Performance Evaluation Corporation (SPEC).

    TPC-C ,TPC-H, and TPC-E are trademarks of the Transaction Performance Processing Council (TPPC).

    VMware, the VMware “boxes” logo and design, Virtual SMP and VMotion are registered trademarks or trademarks (the “Marks”) of VMware, Inc. in the United States and/or other jurisdictions.

    The postings on this site solely reflect the personal views of the author and do not necessarily represent the views, positions, strategies or opinions of IBM or IBM management.

    technorati tags: , , , , ,,,,,,,,,,,,,


    Written by benchmarkingblog

    September 9, 2011 at 3:22 pm

    Posted in Cisco, SPECvirt_sc2010, VMware, x3850

    Tagged with , , ,

    3 Responses

    Subscribe to comments with RSS.

    1. Is IBM working with MSFT on benchmarks with Hyper-V?
      Is IBM working with Oracle on benchmarks with Redhat?
      Is IBM publishing open-standards benchmarks?
      Is IBM publishing leadership benchmarks on Linux/x-86? What is your VM on that platform?

      Tony P

      September 22, 2011 at 2:07 am

    2. Hi Tony – So great to hear from you and hope you are doing well ! IBM has published and is working on benchmarks using many different types of hardware and software. As a start, this site might help –


      September 22, 2011 at 3:39 pm

    3. Anyhow, the VMmark2 benchmark list allows to “bottle -in” any kind of strange configfuration set-ups, from “standard 2 system cluster- which the benchmark needs anyhow since it incorporates the vmotion benchmark function “- to what they call “unmatched pair” , and all kind of other configs.
      SO then next entry could be 8 systems , each a 2-way server, jointed into a 8-system cluster- and becoming just another exmaple of “unmatched pair”.
      Ort use 16 system , each 1 socket, so its a 16-cystem cluster- again “unmatched pair “.

      The real problem is that VMmark2 does not Differentiate enough to sort their result lists properly : this would help to compare apples tp apples and not bananas to fruits.


      October 5, 2011 at 8:12 am

    Leave a Reply

    Fill in your details below or click an icon to log in: Logo

    You are commenting using your account. Log Out /  Change )

    Google+ photo

    You are commenting using your Google+ account. Log Out /  Change )

    Twitter picture

    You are commenting using your Twitter account. Log Out /  Change )

    Facebook photo

    You are commenting using your Facebook account. Log Out /  Change )


    Connecting to %s

    %d bloggers like this: