VMware and Cisco Performance: Fact vs. Fiction

When is a virtualization benchmark really a virtualization benchmark? When is a virtualization benchmark a cloud benchmark?

Cisco has recently claimed that they have achieved “record virtualization performance” on an “industry standard cloud benchmark” with VMmark. Let’s look at the facts:

Fact: As a benchmark, VMmark is ultimately limited by CPU performance and has limited scope for comprehensively measuring overall virtualization. IBM studies consistently show that CPU utilization is often not the performance constraint for x86 virtualization.

Fact: VMmark is certainly not an “industry standard” benchmark as it was not developed through open collaboration by the industry in the manner that TPC or SPEC benchmarks are developed.

Fact: The “world record” result Cisco claims on VMmark needed to use 4 Cisco UCS C460 hosts — not just 2 hosts that have been used in every other VMmark 2.0 published result — and used 160 cores and 320 threads.

Fact: SPECvirt_sc2010 is THE industry-standard virtualization benchmark. SPECvirt_sc2010 measures the end-to-end performance of all system components including the hardware, virtualization platform, and the virtualized guest operating system and application software. IBM is currently #1 in this benchmark.(1) IBM has outstanding results on this virtualization benchmark, whatever virtualization technology is used. Where are the results here from Cisco on this one and on other SPEC and TPC true industry standard benchmarks ?

Fact: VMmark is certainly not a “cloud” benchmark. Cloud complexity is awe-inspiring with its huge scope, variability, and integration. Many organizations such as SPEC are currently working on developing sophisticated cloud benchmarks and will need to focus on consumability metrics such as deployment, elasticity, resiliency, backup, and maintenance.

Calling it virtualization performance does not make it virtualization performance. Calling it cloud does not make it cloud.

************************************************

(1)SPECvirt_sc2010 highwater result on the IBM x3850 X5 (8 Intel Xeon E7-4870, 2.4 GHz, 80 cores, 10 cores/chip, 2 threads/core) with 2TB memory of 7067@432 SPECvirt_sc2010@VMs running Red Hat Enterprise Linux 6.1 (KVM).
Source: http://www.spec.org Results current as of 9/9/11.

SPEC, SPECint, SPECfp, SPECjbb, SPECweb, SPECjAppServer, SPECjEnterprise, SPECjvm, SPECvirt, SPECompM, SPECompL, SPECsfs, SPECpower, SPEC MPI and SPECpower_ssj are trademarks of the Standard Performance Evaluation Corporation (SPEC).

TPC-C ,TPC-H, and TPC-E are trademarks of the Transaction Performance Processing Council (TPPC).

VMware, the VMware “boxes” logo and design, Virtual SMP and VMotion are registered trademarks or trademarks (the “Marks”) of VMware, Inc. in the United States and/or other jurisdictions.

The postings on this site solely reflect the personal views of the author and do not necessarily represent the views, positions, strategies or opinions of IBM or IBM management.

technorati tags: IBM, systems, performance, SPEC, SPECvirt,virtualization,x3850,Cisco,UCS,C460,benchmark,VMware,VMmark,KVM,Xeon,Intel,unified,benchmarking

eX5 Does Mile High TPC-C

I often get tired during a long hike — but nothing a sit down and a few peanut M&Ms can’t take care of. But a few years ago, at about 9000 feet on the edge of a glacier in Montana, that certainly didn’t do the trick. I would walk about three steps, have to sit on a rock, and then I felt as if I wanted to sleep forever. Instead of the usual pondering of how many more miles did we have and how much higher did we have to go, I started to ask deeper questions like why in the world I was doing this at all. The view of the glacier was tremendous but was it worth the feeling that my body had been taken over by aliens?

When I think about “highest ever” I now wonder:

• Is it possible to enjoy hiking in the Rocky Mountains just for the weekend without acclimation?
• When the Junior Olympics are in Denver, do the athletes who make their homes in higher altitudes have an advantage?
• When planning a trip, do I now add altitude to the list of things I need to worry about besides weather, direct flights, crowds, and the language I have to communicate in?

 

But for our “highest ever” this week there are no questions. Just pure celebration. You see, the IBM System x3850 X5 and DB2 just delivered the highest x86-64 performance score ever achieved on the TPC-C benchmark.(1)

The TPC-C benchmark simulates an order-entry environment of a wholesale supplier — entering and delivering orders, recording payments, checking the status of orders, and monitoring the level of stock at the warehouses. TPC-C represents any industry that must manage, sell, or distribute a product or service. Current results show that clients who deploy IBM technology could see more orders entered, faster monitoring, distribution, and delivery.

This configuration used several cool newer technologies — IBM eX5 with E7 processors, MAX5 memory expansion, the amazing DB2 9.7, and SSD storage.

So tell your “highest ever” story here. Keeping it clean, of course.

************************************************

(1) IBM System x®3850 X5 (Intel Xeon E7-8870 processors 2.40GHz, 4 processors/40 cores/80 threads) result of 3,014,684 tpmC, $.59 USD/tpmC, available 9/22/11, DB2 9.7, SUSE Linux® Enterprise Server 11 (SP1),
Source: http://www.tpc.org. Results current as of 8/3/11.

TPC-C ,TPC-H, and TPC-E are trademarks of the Transaction Performance Processing Council (TPPC).

The postings on this site solely reflect the personal views of the author and do not necessarily represent the views, positions, strategies or opinions of IBM or IBM management.

technorati tags: IBM, systems, performance, TPC-C, OLTP,x3850,X5,MAX5,SSD,DB2,benchmark,SUSE,9.7,E7,Xeon,Intel,TPPC,benchmarking