Elisabeth Stahl on Benchmarking and IT Optimization

Will the Real Benchmark Please Stand Up

leave a comment »

They are at it once again. Those imposter benchmarks.

You know. The ones that initially look and feel like real IT industry performance benchmarks.

But then you read the article again, you look a bit more closely and you realize. They are at it again.

So how can we detect and overcome this benchmark fraud ?

  • Make sure the names of the actual benchmarks are clearly stated. You know, something with letters like TPC, SPEC, SAP, STAC, . . .
  • Make sure the metrics are correct. You know, something like transactions per minute or number of users.
  • Make sure there’s a really good footnote with all the details. Just the data is not enough.
  • Make sure there is a link to the site about the benchmark and preferably the results.
  • Make sure that if you sense an imposter benchmark, find REAL data on the systems you are interested in. At an official benchmark or vendor site.  Or run the real workload as a client benchmark.

If you’re not seeing these things, very likely it is some obscure testing that may or may not have a proper benchmark kit, audited results, etc. And it may very likely be artificially tuned to exploit only certain hardware or software that the imposter is looking to promote.

An industry benchmark masquerader that is actually a tried and true swindler.


The postings on this site solely reflect the personal views of the author and do not necessarily represent the views, positions, strategies or opinions of IBM or IBM management.

technorati tags: , , ,,,,,,,,


Written by benchmarkingblog

June 4, 2015 at 6:31 pm

Posted in Uncategorized

Tagged with

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: