The Performance Estimate Low Down

I had avoided it for about as long as I could but I started working on my taxes over the weekend. I thought I might calculate my tax rate but then decided against it. Much too depressing.

There’s been a lot in the news lately on the average tax rate. What is fair, what is not, how to fix it all. Should investment income be taxed at the same rate as your salary? Should Warren Buffett pay the same tax rate as Debbie, his secretary? And does looking at the average of the two make any sense at all?

This discussion reminded me about questions I’ve been getting lately on estimating performance of IT systems.

Systems performance estimates that compare one system to another have sprouted up everywhere. And it has recently come to my attention that many of us have been placing divine reliance on these “performance estimates.” We love to quote them, we use them in many of our capacity and TCO tools, and we may even make huge purchase decisions based on them.

What we need to realize is that sometimes these estimates are based on ridiculously inane models that basically average an OLTP benchmark here and an ISV benchmark there and an HPC benchmark from somewhere else and then throw in something with Java to try to come up with an overall value for a system. Without taking any of the crucial aspects of the technology into consideration. Makes sense, right?

And guess what? Sometimes when there is no input for a certain benchmark in a model, the creator of the performance estimate makes something up. Or even worse, allows the vendor of the system in question to make something up. So if a vendor has published very few benchmarks, most of the performance estimate could be whipped cream.

Almost anything is better than this. So run and measure your workloads for real. Or use a published industry standard or ISV benchmark that matches your workload. Here’s what I’m thinking — it’s imperative to make sure that you understand exactly what is behind every performance estimate that you use. And only ever use them as a last resort.

************************************************

The postings on this site solely reflect the personal views of the author and do not necessarily represent the views, positions, strategies or opinions of IBM or IBM management.

technorati tags: IBM, systems, performance, estimate, benchmark,relative,benchmarking