benchmarkingblog

Benchmarking and Systems Performance

Posts Tagged ‘performance

Avoid “Jump the Gun” Benchmark Tests

leave a comment »

I was talking with someone the other day and noticed something funny. They were chomping at the bit to do some deep down Java tuning. Let’s make ten changes at once and blow this thing out of the water tuning. What they didn’t yet have was a clue on where they were going and how they would even know if they got there.

Before starting any systems performance testing or benchmarking, here are some of my best practices:

  • First things first, define your benchmark objectives. You need success metrics so you know that you have succeeded. They can be response times, they can be transaction rates, they can be users, they can be anything — as long as they are something.
  • Document your hardware/software architecture. Include device names and specifications for systems, network, storage, applications.
  • Implement just one change variable at a time. (OK, sometimes we can get away with a couple.)
  • Keep a change log — what tests were run, what changes were made, what the results were, what your conclusions were for that specific test.
  • Map your tests to what performance reports you based your conclusions on. Sometimes using codes or special syntax when you name your reports helps.
  • Keep going, don’t give up, you will get there.

Some of this we learned in science class. Some of this is common sense. But you’d be surprised sometimes by how much sense these days is uncommon.

************************************************

The postings on this site solely reflect the personal views of the author and do not necessarily represent the views, positions, strategies or opinions of IBM or IBM management.

2NTTAUNP6FGR
technorati tags: , , , , ,,,,,,,

,,,,,,,,

Written by benchmarkingblog

April 13, 2012 at 11:53 am

Posted in Performance, Uncategorized

Tagged with ,

The Performance Estimate Low Down

with 3 comments

I had avoided it for about as long as I could but I started working on my taxes over the weekend. I thought I might calculate my tax rate but then decided against it. Much too depressing.

There’s been a lot in the news lately on the average tax rate. What is fair, what is not, how to fix it all. Should investment income be taxed at the same rate as your salary? Should Warren Buffett pay the same tax rate as Debbie, his secretary? And does looking at the average of the two make any sense at all?

This discussion reminded me about questions I’ve been getting lately on estimating performance of IT systems.

Systems performance estimates that compare one system to another have sprouted up everywhere. And it has recently come to my attention that many of us have been placing divine reliance on these “performance estimates.” We love to quote them, we use them in many of our capacity and TCO tools, and we may even make huge purchase decisions based on them.

What we need to realize is that sometimes these estimates are based on ridiculously inane models that basically average an OLTP benchmark here and an ISV benchmark there and an HPC benchmark from somewhere else and then throw in something with Java to try to come up with an overall value for a system. Without taking any of the crucial aspects of the technology into consideration. Makes sense, right?

And guess what? Sometimes when there is no input for a certain benchmark in a model, the creator of the performance estimate makes something up. Or even worse, allows the vendor of the system in question to make something up. So if a vendor has published very few benchmarks, most of the performance estimate could be whipped cream.

Almost anything is better than this. So run and measure your workloads for real. Or use a published industry standard or ISV benchmark that matches your workload. Here’s what I’m thinking — it’s imperative to make sure that you understand exactly what is behind every performance estimate that you use. And only ever use them as a last resort.

************************************************

The postings on this site solely reflect the personal views of the author and do not necessarily represent the views, positions, strategies or opinions of IBM or IBM management.

technorati tags: , , , , ,,

Written by benchmarkingblog

February 23, 2012 at 2:49 pm

Posted in Performance

Tagged with ,

Dear Performance Advisor Sergio in Austin

leave a comment »

Dear Elisabeth:

I had an experience that I would like to share with your readers.

The other day I had a check engine light show up in my car. To anyone else, this might be a non-issue, I always dread those lights. We all have our limitations, and mine is the inability to resolve any car problem besides an empty fuel tank.

The good news is that after taking my car to the mechanic, the only issue was a loose gas cap that was quickly resolved without charge.

The bad news was that it took a day of inconvenience to find out something that would have been simple to resolve if I had a mechanic as a neighbor.

Although I don’t know anything about working with cars, I do happen to work with a group of IBM experts in Power system performance.

They have recently put together a set of advisors that will monitor current running performance of a live Power system with low overhead.  After monitoring, the advisors will provide a clear understanding of how the system is performing, and provide some expert advice on first places to look for improvements.

Essentially, we have found a way to move a whole team of experts into everyone’s data center.

There are currently 3 advisors that can be downloaded for free from the IBM developerWorks website:

Sergio in Austin

Dear Sergio in Austin:
Thanks so much for your letter. So much better than the column last week entitled “Bride wants to keep friend’s lecherous husband off guest list.” (Yes, this is a real one.) Very exciting news about these wonderful performance tools.  Readers, if you have any questions about them, feel free to send a letter to padvisor@us.ibm.com.  Enjoy.

************************************************

The postings on this site solely reflect the personal views of the author and do not necessarily represent the views, positions, strategies or opinions of IBM or IBM management.

technorati tags: , , , , ,,,,

Written by benchmarkingblog

February 10, 2012 at 5:55 pm

%d bloggers like this: