Elisabeth Stahl on Benchmarking and IT Optimization

Cisco’s March Madness

with 2 comments

I am told that my 5 foot 5 grandmother was a basketball star in high school. The story goes that she racked up many points per game and may have even made some records. I am pretty sure that if she played Jeremy Lin today, she would not be able to keep those records.

The point here is simple: You need to differentiate between adding up records from the beginning of time vs. counting the number of current records.

Last week, Cisco announced new UCS systems, claiming 63 industry benchmark world records overall. Here’s what you need to know:

  • Cisco claims 63 UCS records and provides a link. The link goes to a document which lists 50 proof points. If you’re claiming 63 records, it would be good to see 63 records.
  • Almost a quarter of these “records” are one benchmark and one benchmark only — a virtualization benchmark which only highlights very limited performance ability using a very specific set of products. Not quite the “breadth” that Cisco claims.
  • Some of the results cited are records purely because they were “first to market.” So Cisco was merely first to publish a result on a certain benchmark in a certain category — and may have only been the best for a short while because they were first.
  • Finally, and most importantly, here’s the clincher, so pay really close attention. Cisco’s claimed “records” are “Record as of Publication Date.” This means that Cisco counts records and puts notches in their belt if they have a result that has ever been on top, even for a day over the last three years. The real question is how many of those 63 benchmark results are currently #1 ?   . . .  For many, many years IBM has published web pages with #1 Power benchmarks. With over 100 #1 benchmarks. These over 100 benchmarks are all benchmarks where IBM is currently #1. At the moment in time. Not #1’s culled from all time. I know this. I know this because I personally and painstakingly have to update these very pages.

I would love to see a count of how many records IBM would have if they included #1’s from the beginning of time. I would love to see my grandmother on the court with Jeremy Lin. But I just can’t fathom the awesomeness of either.


The postings on this site solely reflect the personal views of the author and do not necessarily represent the views, positions, strategies or opinions of IBM or IBM management.

technorati tags: , , , , ,,,,,,,,



Written by benchmarkingblog

March 12, 2012 at 3:38 pm

Posted in Cisco

Tagged with , ,

2 Responses

Subscribe to comments with RSS.

  1. I would suggest that given the massive adoption of UCS worldwide that IBM spend more time focusing on why they are losing market share than bashing others over records.

    Ryan Watters

    March 13, 2012 at 4:21 pm

  2. @Ryan From a benchmark practitioner’s stance, Elisabeth is right to criticize the claim of “63 world record performance benchmarks”. The Cisco definition of “world record” here would assign a permanent gold medal to any result that is (or was) momentarily highest. The tally also includes fence claims (e.g. “best 2-socket x86 result”, which isn’t the best overall result).

    Using Cisco’s defintion of “world record”, I count about 250 world records among the ~300 currently published TPC-C results.

    There are some current Cisco UCS benchmark results which do surpass other servers; Cisco deserves credit for those. They also deserve plaudits any time they publish a “first to market” result. And I see nothing wrong with a heavy mix of VMmark results — it’s certainly a “hot” x86 benchmark today.

    Daniel Bowers

    March 15, 2012 at 11:35 am

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: