Elisabeth Stahl on Benchmarking and IT Optimization

The Performance Estimate Low Down

with 3 comments

I had avoided it for about as long as I could but I started working on my taxes over the weekend. I thought I might calculate my tax rate but then decided against it. Much too depressing.

There’s been a lot in the news lately on the average tax rate. What is fair, what is not, how to fix it all. Should investment income be taxed at the same rate as your salary? Should Warren Buffett pay the same tax rate as Debbie, his secretary? And does looking at the average of the two make any sense at all?

This discussion reminded me about questions I’ve been getting lately on estimating performance of IT systems.

Systems performance estimates that compare one system to another have sprouted up everywhere. And it has recently come to my attention that many of us have been placing divine reliance on these “performance estimates.” We love to quote them, we use them in many of our capacity and TCO tools, and we may even make huge purchase decisions based on them.

What we need to realize is that sometimes these estimates are based on ridiculously inane models that basically average an OLTP benchmark here and an ISV benchmark there and an HPC benchmark from somewhere else and then throw in something with Java to try to come up with an overall value for a system. Without taking any of the crucial aspects of the technology into consideration. Makes sense, right?

And guess what? Sometimes when there is no input for a certain benchmark in a model, the creator of the performance estimate makes something up. Or even worse, allows the vendor of the system in question to make something up. So if a vendor has published very few benchmarks, most of the performance estimate could be whipped cream.

Almost anything is better than this. So run and measure your workloads for real. Or use a published industry standard or ISV benchmark that matches your workload. Here’s what I’m thinking — it’s imperative to make sure that you understand exactly what is behind every performance estimate that you use. And only ever use them as a last resort.


The postings on this site solely reflect the personal views of the author and do not necessarily represent the views, positions, strategies or opinions of IBM or IBM management.

technorati tags: , , , , ,,


Written by benchmarkingblog

February 23, 2012 at 2:49 pm

Posted in Performance

Tagged with ,

3 Responses

Subscribe to comments with RSS.

  1. Thanks for keeping us straight on performance comparisons!

    Phil Hartman

    February 29, 2012 at 12:26 pm

  2. […] server performance, I highly recommend  you check the benchmarkingblog and articles such as The Performance Estimate Low Down and Performance in a Flash: New IBM XIV SSD Caching. Share […]

  3. Is this how you would also describe IBM’s repeated use of IDEAS International’s RPE2 (Relative Performance Estimate) for sales and marketing?

    D Burrows

    March 9, 2012 at 10:28 am

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: