Avoid “Jump the Gun” Benchmark Tests
I was talking with someone the other day and noticed something funny. They were chomping at the bit to do some deep down Java tuning. Let’s make ten changes at once and blow this thing out of the water tuning. What they didn’t yet have was a clue on where they were going and how they would even know if they got there.
Before starting any systems performance testing or benchmarking, here are some of my best practices:
- First things first, define your benchmark objectives. You need success metrics so you know that you have succeeded. They can be response times, they can be transaction rates, they can be users, they can be anything — as long as they are something.
- Document your hardware/software architecture. Include device names and specifications for systems, network, storage, applications.
- Implement just one change variable at a time. (OK, sometimes we can get away with a couple.)
- Keep a change log — what tests were run, what changes were made, what the results were, what your conclusions were for that specific test.
- Map your tests to what performance reports you based your conclusions on. Sometimes using codes or special syntax when you name your reports helps.
- Keep going, don’t give up, you will get there.
Some of this we learned in science class. Some of this is common sense. But you’d be surprised sometimes by how much sense these days is uncommon.
************************************************
The postings on this site solely reflect the personal views of the author and do not necessarily represent the views, positions, strategies or opinions of IBM or IBM management.
2NTTAUNP6FGR
technorati tags: IBM, systems, performance, benchmarking, variable,log,hardware,query,queries,power,benchmark,
software,database,machine,architecture,change,tuning,BI,practices,
Expert Integrated Systems: A Revolutionary Repast
This past weekend I needed to make dinner for 15 people. Twice.
I organized my shopping into two trips, not ten. I cooked one side dish but bought another. One night I grilled, one I roasted. I set the table ahead of time. I admit it — I even bought bagged salad.
The point here is simplification. Do what we need to with limited resources. Save time, reduce costs, get the performance we require. And that’s one of the main points of Expert Integrated Systems.
With new IBM PureSystems, you get the best of all worlds. A combination of the flexibility of a general purpose system with the simplicity of an appliance. It’s revolutionary and evolutionary. And all of this with outstanding performance, of course:
- View the excellent results on the SAP SD (Sales and Distribution) benchmark on the new IBM Flex System p460 and x240 with DB2. (1)
- Analyze the #1 x86 16-core TPC-C result on the IBM Flex System x240. (2)
- Read the press release, watch the video.
- See more great benchmark results.
- Peruse the rPerf data for the IBM Flex System p260 and p460.
- Follow IBM PureSystems.
Performance plus powerful systems management capabilities. Game-changing technology. Like my automatic dishwasher, which was also quite popular this weekend, and luckily performed quite well.
************************************************
(1) IBM Flex System p460 – Source: sap.com/benchmark. Configuration and results of the IBM Flex System p460 on the two-tier SAP SD standard application benchmark running SAP enhancement package 4 for SAP ERP 6.0 (Unicode): 4 processors /32 cores / 128 threads, POWER7, 3.55 GHz, 256 GB memory, 17,000 SAP SD benchmark users, dialog response: 0.96 seconds, line items/hour: 1,861,670, dialog steps/hour: 5,585,000, SAPS: 93,080, database response time (dialog/update): 0.011 sec / 0.021 sec, CPU utilization: 99%, OS: AIX 7.1, DB2 9.7; Certification #2012015.
IBM Flex System x240 – Source: sap.com/benchmark. Configuration and results of the IBM Flex System x240 on the two-tier SAP SD standard application benchmark running SAP enhancement package 4 for SAP ERP 6.0 (Unicode): 2 processors /16 cores / 32 threads, Intel Xeon Processor E5-2690, 2.90 GHz, 64 KB L1 cache and
256 KB L2 cache per core, 20 MB L3 cache per processor,
128 GB main memory, 7,960 SAP SD benchmark users, dialog response: 0.98 seconds, line items/hour: 870,330, dialog steps/hour: 2,611,000, SAPS: 43,520, database response time (dialog/update): 0.012 sec / 0.010 sec, CPU utilization: 99%, OS: Windows Server 2008 R2 Enterprise Edition, DB2 9.7; Certification #2012016.
Results valid as of 4/11/2012.
(2)TPC-C result on IBM Flex System x240, Performance (tpmC): 1,503,544, Price/Performance: .53 USD, Pro! cessors / MHz of Server: Intel Xeon Processor E5-2690 2.90GHz / 2.90GHz
Total # of Processors / Cores / Threads: 2 / 16 / 32, Availability Date: 8/16/2012. Result current as of 4/11/12. Source: www.tpc.org
SAP and all SAP logos are trademarks or registered trademarks of SAP AG in Germany and in several other countries.
TPC-H, TPC-C, and TPC-E are trademarks of the Transaction Performance Processing Council (TPPC).
The postings on this site solely reflect the personal views of the author and do not necessarily represent the views, positions, strategies or opinions of IBM or IBM management.
technorati tags: IBM, systems, performance, SPEC, PureAnalytics,PureDatabase,Flex,NGP,POWER7,expert,
benchmark,Intel,integrated,SAP,rPerf,TPC-C,SD,PureSystems,Intel,PureFlex,
p460,p260,DB2,appliance,Troy,Power,PureApplication,benchmarking
Is Oracle Fleecing You?
A couple of days ago, I bought a new fleece hiking jacket. In small, in black, on sale. With a hood. Made by a high end manufacturer whose name evokes a part of a mountain. The plush is luxuriously thick, the zipper hardware is incredible, and the tailoring just feels so right. I had tried on other jackets that just did not stack up. There was no comparison, apples and oranges, from different planets. And that reminded me once again of how some performance comparisons are made.
This week, Oracle claimed x86 “world-record” performance with the Sun Fire X4800 M2 on industry standard Java middleware and transactional database benchmarks. They compared their results to results from IBM. Here’s what you need to know:
For the Java SPECjEnterprise2010 benchmark, Oracle needed more cores in both the application server and the database server. Oracle used over 4x the storage disks and over 7x the cache that was used in the IBM result. Oracle conveniently cites price/performance (which isn’t even a metric in this benchmark) for the application tier only. The picture would probably look very different if they included the important database tier (with all those costly Oracle licenses). Oracle compares their brand new result with an IBM result from over a year ago.(1) Maybe it’s time for a new . . .
For the transactional TPC-C benchmark, when you analyze the comparison correctly, the IBM result is actually 19% better performance per core than the Oracle result. The IBM configuration has been available for months, the Oracle configuration is not even available. And the Oracle result is 1.5x more expensive.(2)
When I got to the checkout line with my perfect fleece jacket, they told me that the store was having a special one day sale, another 20% off. I got myself superior performance and price/performance — and you can’t get much better than that.
************************************************
(1) WebSphere Application Server V7 on IBM Power 780 and DB2 on IBM Power 750 Express, (64 core app server, 32 core db server), 16,646.34 SPECjEnterprise2010 EjOPS vs. Oracle WebLogic Server 12c and Oracle Database 11g Release 2 with Oracle Linux running on a Sun Fire X4800 M2 server(5U) with eight Intel Xeon E7-8870 2.4 GHz processors, (80 cores, 8 chips, 10 cores/chip, 2 threads/core) 27,150.05 SPECjEnterprise2010 EjOPS.
(2) IBM System x3850 X5 (4 chips/40 cores/80 threads) – 3,014,684 tpmC, US$.59/tpmC, available 09/22/11 vs. Oracle Sun Fire X4800 M2 server (8 chips/80 cores/160 threads) – 5,055,888 tpmC, US$.89/tpmC, available 06/26/12.
Sources: www.tpc.org, www.spec.org. Results current as of 3/30/12.
TPC-C ,TPC-H, and TPC-E are trademarks of the Transaction Performance Processing Council (TPPC).
SPEC, SPECint, SPECfp, SPECjbb, SPECweb, SPECjAppServer, SPECjEnterprise, SPECjvm, SPECvirt, SPECompM, SPECompL, SPECsfs, SPECpower, SPEC MPI and SPECpower_ssj are trademarks of the Standard Performance Evaluation Corporation (SPEC).
The postings on this site solely reflect the personal views of the author and do not necessarily represent the views, positions, strategies or opinions of IBM or IBM management.
technorati tags: IBM, systems, performance, SPEC, SPECjEnterprise,TPC-C,780,750,POWER7,WebSphere,benchmark,Xeon,11g,WebLogic,M4,E7-8870,
More March Madness with Oracle Exadata
There’s a basketball play called the elbow pass. The elbow pass is a behind-the-back pass — as the ball crosses the player’s back, the player hits it with his elbow, redirecting the ball back toward the side it started on. It is rarely used and pretty much never results in points or a win.
This morning, Oracle once again advertised on the front page of the Wall Street Journal. The ad claimed that Exadata is faster. Here is what you need to know on this one:
- The ad states “up to 20x faster queries.” There is no data. But if we assume this is true then this could mean that Oracle may have ONE query that is faster. IBM could surpass Oracle on every other query. Sort of like if you are only good at the obscure elbow pass.
- If the European client that the claim is totally based on is so pleased with Oracle, why are they not allowing Oracle to use their name? Wow, a ghost reference.
- What system is this even being compared with?
- If Oracle’s database performance is so great on Exadata, where are the Exadata industry standard benchmarks? Suffice it to say that we’ll probably see an elbow pass before we see one of those.
************************************************
The postings on this site solely reflect the personal views of the author and do not necessarily represent the views, positions, strategies or opinions of IBM or IBM management.
technorati tags: IBM, systems, performance, Oracle, Exadata,Europe,retail,query,queries,power,benchmark,AIX,
Cisco’s March Madness
I am told that my 5 foot 5 grandmother was a basketball star in high school. The story goes that she racked up many points per game and may have even made some records. I am pretty sure that if she played Jeremy Lin today, she would not be able to keep those records.
The point here is simple: You need to differentiate between adding up records from the beginning of time vs. counting the number of current records.
Last week, Cisco announced new UCS systems, claiming 63 industry benchmark world records overall. Here’s what you need to know:
- Cisco claims 63 UCS records and provides a link. The link goes to a document which lists 50 proof points. If you’re claiming 63 records, it would be good to see 63 records.
- Almost a quarter of these “records” are one benchmark and one benchmark only — a virtualization benchmark which only highlights very limited performance ability using a very specific set of products. Not quite the “breadth” that Cisco claims.
- Some of the results cited are records purely because they were “first to market.” So Cisco was merely first to publish a result on a certain benchmark in a certain category — and may have only been the best for a short while because they were first.
- Finally, and most importantly, here’s the clincher, so pay really close attention. Cisco’s claimed “records” are “Record as of Publication Date.” This means that Cisco counts records and puts notches in their belt if they have a result that has ever been on top, even for a day over the last three years. The real question is how many of those 63 benchmark results are currently #1 ? . . . For many, many years IBM has published web pages with #1 Power benchmarks. With over 100 #1 benchmarks. These over 100 benchmarks are all benchmarks where IBM is currently #1. At the moment in time. Not #1′s culled from all time. I know this. I know this because I personally and painstakingly have to update these very pages.
I would love to see a count of how many records IBM would have if they included #1′s from the beginning of time. I would love to see my grandmother on the court with Jeremy Lin. But I just can’t fathom the awesomeness of either.
************************************************
The postings on this site solely reflect the personal views of the author and do not necessarily represent the views, positions, strategies or opinions of IBM or IBM management.
technorati tags: IBM, systems, performance, Cisco, SPEC,TPC,SAP,VMware,VMmark,UCS,benchmark,Xeon,
A Better Future — and Much Faster with IBM
Is the future always better? Sometimes it’s not so clear.
Take reading. Now that everyone is reading on electronic devices, we complain about getting distracted by email and tweets. No wonder we’re not finishing Anna Karenina.
Take brushing teeth. I read this morning that kids these days are requiring oral surgery to fill numerous cavities — because they are now imbibing more juice drinks and parents are not making them brush their teeth because the kids don’t like brushing their teeth. Cry now or really cry later.
Take voting. Since moving to Ohio, I get hundreds of calls the day before Super Tuesday. Now they’re all automated so I can’t even hang up on anybody.
And sometimes our frustrations with the future should just be authored as a First World Problems meme.
Today there is something really exciting for our future. The new IBM System x and BladeCenter systems with Intel Xeon Processor E5-2600 series processors were just announced.
And with them, there were many wonderful benchmarks, outstanding leadership proof points from IBM including OLTP (TPC-E), SAP SD Two-Tier Standard Application Benchmark, virtualization (SPECvirt_sc2010), Java (SPECjbb2005), energy efficiency (SPECpower_ssj2008) and high performance computing (SPEC CPU2006). A whole portfolio of diverse workloads.
Making Smarter Computing real.
And a much faster future.
************************************************
Sources: www.tpc.org, www.spec.org, www.sap.com/benchmark.
TPC-C ,TPC-H, and TPC-E are trademarks of the Transaction Performance Processing Council (TPPC).
SPEC, SPECint, SPECfp, SPECjbb, SPECweb, SPECjAppServer, SPECjEnterprise, SPECjvm, SPECvirt, SPECompM, SPECompL, SPECsfs, SPECpower, SPEC MPI and SPECpower_ssj are trademarks of the Standard Performance Evaluation Corporation (SPEC).
SAP, mySAP and other SAP product and service names mentioned herein as well as their respective
logos are trademarks or registered trademarks of SAP AG in Germany and in several other countries all
over the world.
The postings on this site solely reflect the personal views of the author and do not necessarily represent the views, positions, strategies or opinions of IBM or IBM management.
technorati tags: IBM, systems, performance, SPEC, SPECjbb,TPC-E,SAP,SPECvirt,E5,SPECpower,benchmark,Xeon,11g,SPECcpu,
The Performance Estimate Low Down
I had avoided it for about as long as I could but I started working on my taxes over the weekend. I thought I might calculate my tax rate but then decided against it. Much too depressing.
There’s been a lot in the news lately on the average tax rate. What is fair, what is not, how to fix it all. Should investment income be taxed at the same rate as your salary? Should Warren Buffett pay the same tax rate as Debbie, his secretary? And does looking at the average of the two make any sense at all?
This discussion reminded me about questions I’ve been getting lately on estimating performance of IT systems.
Systems performance estimates that compare one system to another have sprouted up everywhere. And it has recently come to my attention that many of us have been placing divine reliance on these “performance estimates.” We love to quote them, we use them in many of our capacity and TCO tools, and we may even make huge purchase decisions based on them.
What we need to realize is that sometimes these estimates are based on ridiculously inane models that basically average an OLTP benchmark here and an ISV benchmark there and an HPC benchmark from somewhere else and then throw in something with Java to try to come up with an overall value for a system. Without taking any of the crucial aspects of the technology into consideration. Makes sense, right?
And guess what? Sometimes when there is no input for a certain benchmark in a model, the creator of the performance estimate makes something up. Or even worse, allows the vendor of the system in question to make something up. So if a vendor has published very few benchmarks, most of the performance estimate could be whipped cream.
Almost anything is better than this. So run and measure your workloads for real. Or use a published industry standard or ISV benchmark that matches your workload. Here’s what I’m thinking — it’s imperative to make sure that you understand exactly what is behind every performance estimate that you use. And only ever use them as a last resort.
************************************************
The postings on this site solely reflect the personal views of the author and do not necessarily represent the views, positions, strategies or opinions of IBM or IBM management.
technorati tags: IBM, systems, performance, estimate, benchmark,relative,benchmarking
The “Look what we got!” email
Yesterday morning at 7AM I received an email from the unlikeliest of sources. My mother.
You may wonder why receiving an email from my mother would be unusual. The startling fact is: My mother had never owned a computer before.
My mother does everything at the mall. My mother shops, of course, at the mall. My mother eats at the mall. My mother even exercises at the mall (when the weather is bad).
So my mother went out the other day and bought a computer at the mall. She bought a printer at the mall. She even bought internet access at the mall.
My mother talks about “apps” now. She talks “wireless.” She emails at 7AM.
My mother is for this week only at the very forefront of technology. When I called her yesterday she said that her icons were doing something funny. Luckily, my mother also bought a support contract at the mall — so if she has operational problems or even performance challenges, at least she won’t be calling me.
“Sent from my iPad” next to my mother’s name. Still not computing. In her words, “We have joined the 21st century. I hope we can figure this thing out.”
___________________________________________________________
The postings on this site solely reflect the personal views of the author and do not necessarily represent the views, positions, strategies or opinions of IBM or IBM management.
technorati tags: IBM, systems, performance, iPad, mother,benchmarking
Dear Performance Advisor Sergio in Austin
Dear Elisabeth:
I had an experience that I would like to share with your readers.
The other day I had a check engine light show up in my car. To anyone else, this might be a non-issue, I always dread those lights. We all have our limitations, and mine is the inability to resolve any car problem besides an empty fuel tank.
The good news is that after taking my car to the mechanic, the only issue was a loose gas cap that was quickly resolved without charge.
The bad news was that it took a day of inconvenience to find out something that would have been simple to resolve if I had a mechanic as a neighbor.
Although I don’t know anything about working with cars, I do happen to work with a group of IBM experts in Power system performance.
They have recently put together a set of advisors that will monitor current running performance of a live Power system with low overhead. After monitoring, the advisors will provide a clear understanding of how the system is performing, and provide some expert advice on first places to look for improvements.
Essentially, we have found a way to move a whole team of experts into everyone’s data center.
There are currently 3 advisors that can be downloaded for free from the IBM developerWorks website:
Sergio in Austin
Dear Sergio in Austin:
Thanks so much for your letter. So much better than the column last week entitled “Bride wants to keep friend’s lecherous husband off guest list.” (Yes, this is a real one.) Very exciting news about these wonderful performance tools. Readers, if you have any questions about them, feel free to send a letter to padvisor@us.ibm.com. Enjoy.
************************************************
The postings on this site solely reflect the personal views of the author and do not necessarily represent the views, positions, strategies or opinions of IBM or IBM management.
technorati tags: IBM, systems, performance, PowerVM, JAVA,VIOS,advisor,POWER7,benchmarking
Performance in a Flash: New IBM XIV SSD Caching
3 ways that caching can help performance:
Caching is a type of animal behavior where an animal will store food in times of surplus for times when food is less plentiful. The place where the food is stored is called a cache and it is hidden from competing individuals. Caching can be on a long-term basis – cached on a seasonal cycle, with food to be consumed months down the line. Think acorns and squirrels. Where I live, caching is what gets a squirrel through the long cold snowy winter. So you see, caching helps performance. Geocaching is an outdoor sporting activity, hiding and seeking containers, called “caches.” This “game of high-tech hide and seek” can involve toys and Tupperware. I helped an 8 year old find this treasure long before one relied on a GPS to tell you exactly where the cache was. The best part of this activity, in my opinion, was the extremely long walks on mid summer evenings, sometimes in gigantic mile-wide circles, to find what was hidden. I was never so fit. So you see, caching helps performance. IBM has just announced XIV with SSD Caching. High capacity SSDs are used as secondary cache (400GB SSD device can be added to each module, 6TB of cache per rack, scales with the system – 6 to 15 SSD drives). No tuning necessary. And the very best part — nearly 3X IOPS increase for OLTP workloads. And other amazingly impressive performance feats.
So you really see, caching helps performance.
************************************************
The postings on this site solely reflect the personal views of the author and do not necessarily represent the views, positions, strategies or opinions of IBM or IBM management.
technorati tags: IBM, systems, performance, SSD, XIV,cache,caching,OLTP,volume,controller,benchmark,storage,IOPS,SPC,ASU,storage,flash,benchmarking
