benchmarkingblog

Elisabeth Stahl on Benchmarking and IT Optimization

Archive for the ‘TPC-H’ Category

Come on Oracle, Get “With It” Benchmarking

leave a comment »

I admit that many weekends this time of year you will find me (when I’m not enslaved by the leaf blower) curled up with a good old book on the old couch with my thankfully not so old dog.

But this weekend I truly was “with it.”

On Saturday night I attended one of the most sought-after sold out concert events ever to hit this town. I got to see a Pink concert that included not only 17 of her best songs but Pink flying through the air doing acrobatics that you simply would not believe. A rock concert rolled right into the circus, truly amazing.

And then to top it all off, on Sunday night I attended one of the most sought-after sold out movie events of the year. I got to see the latest Hunger Games, Catching Fire, second in this awesome trilogy which could be even more popular than the first.

So on Monday when I saw the latest Oracle SPARC T5-4 benchmark result on the TPC-H decision support benchmark (1), all I could think was how so “not with it.”

Like Gangnam Style this year. Or What Does the Fox Say this month.

Hey, I’m the first one to like legacy. My closet is filled with vintage looks. I love retro — just not when it comes to benchmarks.

Here is what you need to know.

  • First of all, this is TPC-H. Yawn. We’re ready for something new here.
  • Most of the TPC-H results are grayed out in this category, considered “historic.” This result is right next to a result from IBM — from 2007 (yes you heard that right).
  • Total Storage to Database Size ratio is a massive 60.80. Talk about overkill on storage to achieve performance. This number is many many times the ratio we’ve seen from other results.
  • Load time is a whopping 9.63 hours.
  • 128 query streams are needed. Most results use many, many fewer. That’s because TPC-H has a limited number of query variations; so when you run a lot of streams, you have a high probability that the same queries will be requested more than once. Oracle is greatly increasing the probability that they will have the results of the queries stored in their cache — which may not be representative of how their product would perform in a truly ad hoc query environment.
  • Oracle once again included extremely minimal support in their pricing. Does $2300 a year sound like what you are paying for software support?

************************************************
(1) Oracle TPC-H of 377,594 QphH@10000GB,$4.65 per QphH,Availability 11/25/13,Oracle Database 11g R2 Enterprise Edition w/Partitioning,SPARC T5 3.6 GHz; Total # of Processors: 4,Total # of Cores: 64,Total # of Threads: 512.
Source: http://www.tpc.org. Results current as of 11/25/13.

TPC-C ,TPC-H, and TPC-E are trademarks of the Transaction Performance Processing Council (TPPC).

The postings on this site solely reflect the personal views of the author and do not necessarily represent the views, positions, strategies or opinions of IBM or IBM management.

technorati tags: , , ,,,,,,,,,,,,,

,,

Written by benchmarkingblog

November 26, 2013 at 11:28 am

Posted in SPARC T5, TPC-H

Tagged with , ,

The National Security on the T5-4 and Big Data

with 5 comments

There’s been a lot of talk the last few days on Big Data and when it’s “right” to capture and use it. Some say it’s a real invasion of privacy. Others realistically point out that it is the best way to counter terrorism.

Whichever you believe, the important thing is that Big Data is being discussed not just in geeky meetings with IT managers but by everybody. When your neighbor across the street stops trimming his tree branches just to talk to you about it, you know it’s hot stuff.

So I was particularly interested to see that Oracle just published a new TPC-H data benchmark result on the SPARC T5-4.

And here is what hits you like a train.

  • Why is this publish at only the 3TB size when all the talk these days is on much larger amounts of data?
  • Why is the Total Storage to Database Size ratio a whopping 29? Talk about overkill on storage to achieve performance. This number is many times the ratio we’ve seen from other results.
  • Why is the memory to database size % a whopping 66.6? Again, much more than you should need and what we normally see.
  • Why are there 192 query streams needed? Most results use many, many fewer. That’s because TPC-H has a limited number of query variations; so when you run a lot of streams, you have a high probability that the same queries will be requested more than once. Oracle is greatly increasing the probability that they will have the results of the queries stored in their cache — which may not be representative of how their product would perform in a truly ad hoc query environment.
  • Why isn’t the configuration available now? Because key elements of the storage are not ready.
  • Why did Oracle once again include extremely minimal support in their pricing? Does $2300 a year sound like what you are paying for software “incident server support” . . . ? You don’t even need to answer this one.

Comments are welcome at your own risk.

************************************************
(1) Oracle TPC-H of 409,721 QphH@3000GB,$3.94 per QphH,Availability 09/24/13,Oracle Database 11g R2 Enterprise Edition w/Partitioning,SPARC T5 3.6 GHz; Total # of Processors: 4,Total # of Cores: 64,Total # of Threads: 512.
Source: http://www.tpc.org. Results current as of 6/12/13.
TPC-C ,TPC-H, and TPC-E are trademarks of the Transaction Performance Processing Council (TPPC).

The postings on this site solely reflect the personal views of the author and do not necessarily represent the views, positions, strategies or opinions of IBM or IBM management.

technorati tags: , , ,,,,,,,,,,,,,,

,

Written by benchmarkingblog

June 12, 2013 at 3:36 pm

Posted in SPARC T5, TPC-H

Tagged with , , ,

I Want to Buy a Zoo, Not an Oracle System

with 5 comments

Most people don’t think about going to the zoo in the middle of winter. But it’s actually the very best time to go.

No crowds, the zebra fried oreo shacks are closed, and the animals are at their very best. And, after all, isn’t that what it’s really all about?

Last week, I had a spectacular visit. The polar bears were playing with dead Christmas trees, the seals and sea lions were playing with their rubber balls, and the grizzly cubs were playing with themselves.

And then there’s the Rainforest. Imagine stepping from a cold snowy Cleveland day into a zoological tropical paradise. Sort of like the feeling you get when the plane doors open, you get your carry-on from the overhead, and you step off in Miami. My rainforest favorites are the tropical monkeys, the river otters, and, of course, the anteaters. But alas, last week, the anteaters were still, lying on their sides, hiding behind a crop of rocks.

And that’s how I’ve been feeling lately about Oracle.

  • See what’s hiding behind the Oracle SPARC SuperCluster and Exadata systems. “Must buy” storage server software.
  • See what’s hiding behind that pricing in the Oracle benchmark. Artificially low support costs.
  • See what’s hiding behind Oracle’s “Itanium roadmap.” An investigation into Oracle’s “potentially abusive” practices.
  • What else do YOU think Oracle is hiding? All thoughts welcome.

    ************************************************

    Oracle SPARC T4-4 server (4 sockets/32 cores/256 threads) 205,792 QphH@3000GB, $4.10/QphH@3000GB, available 5/31/12.
    Source: http://www.tpc.org. Results current as of 1/11/12.
    TPC-C ,TPC-H, and TPC-E are trademarks of the Transaction Performance Processing Council (TPPC).

    The postings on this site solely reflect the personal views of the author and do not necessarily represent the views, positions, strategies or opinions of IBM or IBM management.

    technorati tags: , , , , ,,,,,,,,,,,,,

    Written by benchmarkingblog

    January 11, 2012 at 1:41 pm

    All I Want for Christmas is a Good T4 Benchmark

    with 7 comments

    I read an article the other day about a Santa school where aspiring Santas go to learn about how to be Santa Claus. Important concepts like how to comb your beard, how to make that amazing Ho Ho Ho, and where to properly keep your hands when a kid is on your lap.

    In this economy one of the new lessons that the Santa school offers is on how to reduce kids’ expectations. Santa’s workshop has too many toys to produce Johnny, Santa can’t fit everything into his sack, a bunch of elves got sick.

    Oracle announced a new TPC-H BI 3TB benchmark result today on the Oracle SPARC T4-4 server. Expectations surely need to be reduced on this one because here’s what you need to know:

    • Oracle once again used an embarrassing amount of storage to run this benchmark — almost three times the amount of storage that IBM used.(1)
    • Oracle once again used 128 query streams for this benchmark compared with IBM’s 8. TPC-H has a limited number of query variations; so when you run a lot of streams, you have a high probability that the same queries will be requested more than once. Oracle is greatly increasing the probability that they will have the results of the queries stored in their cache — which may not be representative of how their product would perform in a truly ad hoc query environment.
    • Oracle once again published a benchmark where the configuration isn’t even available — until May 31, 2012.
    • Oracle once again included extremely minimal support in their pricing. Does $2300 a year sound like what you are paying for software “incident server support” . . . ?

     

     

    The most important lesson for Santas — Never promise anything.

    ************************************************

    (1)IBM POWER 780 (8 sockets/32 cores/128 threads) 192,001 QphH@3000GB, $6.37/QphH@3000GB, available 11/30/11, 15,610 GB storage. Oracle SPARC T4-4 server (4 sockets/32 cores/256 threads) 205,792 QphH@3000GB, $4.10/QphH@3000GB, available 5/31/12, 45,600 GB storage.
    Source: http://www.tpc.org. Results current as of 11/30/11.
    TPC-C ,TPC-H, and TPC-E are trademarks of the Transaction Performance Processing Council (TPPC).

    The postings on this site solely reflect the personal views of the author and do not necessarily represent the views, positions, strategies or opinions of IBM or IBM management.

    technorati tags: , , , , ,,,,,,,,,,,,,

    Written by benchmarkingblog

    November 30, 2011 at 11:14 am

    Posted in Oracle, POWER7, SPARC T4, TPC-H

    Tagged with , , , , ,

    Is There a “Right” and “Wrong” with Benchmarks?

    with 4 comments

    Like many of our politicians lately, Oracle has been proselytizing — in this case that there are certain benchmarks which are right and certain benchmarks which are wrong.

    In their evaluation Oracle claims to focus on all the right benchmarks while claiming that everyone else is focusing on the wrong ones. Let’s take a look at their reviews:

  • Thumbs up for SPECjEnterprise – Yes, I wholeheartedly agree, a good benchmark to use for Java applications. And that’s why IBM has outstanding results on this benchmark. Oracle’s T4 result needed four times the number of app nodes, twice the number of cores, almost four times the amount of memory and significantly more storage than the IBM POWER7 result.(1)
  • Thumbs up for Oracle ebusiness, JDEdwards, PeopleSoft, Siebel – Good when you need to model these exact applications but keep in mind that Oracle owns these “independent” benchmarks. Even with a stacked deck, IBM has #1’s here.
  • Thumbs up for TPC-H – Yes, totally agree that this benchmark is valuable for business intelligence applications. Which is why IBM just published a leadership 3TB TPC-H result.(2) And Oracle’s T4 result needed 2.7x the amount of storage and 14x the number of streams than the IBM POWER7 1TB result.(3)
  • Thumbs up for SPECweb2005 – Funny that Oracle likes this one since there is a definite lack of publishes here anymore and this benchmark is being retired in January.
  • Thumbs up for TPC-C – Even funnier that Oracle likes this one since Sun avoided publishing on this for many, many years claiming this benchmark was too old. Now, as a fair-weather friend, it’s a favorite. IBM has the top nonclustered result (even with an older Power system).(4)
  • Thumbs down for SAP SD – Interesting that this valuable benchmark for SAP users is on Oracle’s hit list. This benchmark is a wonderful measure for SAP users and has many proof points from many vendors. Is that why Oracle tries to avoid this one and instead publishes on other more obscure SAP benchmarks like SD-Parallel and ATO?
  • Thumbs down for SPECjAppServer2004 – Looking to fill up space, since Oracle knows as well as we do that this benchmark was retired almost a year ago. Also ironic because I do seem to remember that Oracle/Sun used to really like this benchmark. Alot.
  • Thumbs down for SPECcpu and SPECjbb – These are nice and easy for integer, floating point, and Java business apps. Surely valuable in their own way.
  • Thumbs down for Stream – Ignoring HPC users are we? We’ve seen this before with the TOP500. IBM has the most systems with 213. Oracle had 12.
  • Keep in mind that Oracle has only published their latest SPARC T4 results on 2 industry standard benchmarks — that’s right, only 2. Which is fine, if you don’t mind steering with your eyes closed.

    My view is that you may want and need to look at different aspects of systems across a wide portfolio of many different types of benchmarks. Which is why IBM is #1 — in over 100.

    ************************************************

    (1)Oracle WebLogic Server 11g and Oracle Database 11g Release 2 with Oracle Real Application Clusters and Oracle Solaris running on a four-node SPARC T4-4 cluster, each system with four SPARC T4 3GHz processors, (128 core app server, 64 core db server), 40,104.86 SPECjEnterprise2010 EjOPS vs. WebSphere Application Server V7 on IBM Power 780 and DB2 on IBM Power 750 Express, (64 core app server, 32 core db server), 16,646.34 SPECjEnterprise2010 EjOPS.
    (2)IBM POWER 780 (8 sockets/32 cores/128 threads) 192,001 QphH@3000GB, $6.37/QphH@3000GB, available 11/30/11.
    (3)SPARC T4-4 server (4 sockets/32 cores/256 threads) 201,487 QphH@1000GB, $4.60/QphH@1000GB, available 10/30/11. IBM POWER 780 Model 9179-MHB server (8 sockets/32 cores/128 threads) 164,747.2 QphH@1000GB, $6.85/QphH@1000GB, available 3/31/11.
    (4)A 64-core IBM Power 595 (5 GHz, 32 chips, 128 threads) with IBM DB2 9.5 is the best overall single system (6,085,166 tpmC, $2.81/tpmC, configuration available 12/10/08).
    Sources: http://www.spec.org, http://www.tpc.org, http://www.top500.org. Results current as of 10/26/11.

    TPC-C ,TPC-H, and TPC-E are trademarks of the Transaction Performance Processing Council (TPPC).

    SAP, mySAP and other SAP product and service names mentioned herein as well as their respective
    logos are trademarks or registered trademarks of SAP AG in Germany and in several other countries all
    over the world.

    SPEC, SPECint, SPECfp, SPECjbb, SPECweb, SPECjAppServer, SPECjEnterprise, SPECjvm, SPECvirt, SPECompM, SPECompL, SPECsfs, SPECpower, SPEC MPI and SPECpower_ssj are trademarks of the Standard Performance Evaluation Corporation (SPEC).

    The postings on this site solely reflect the personal views of the author and do not necessarily represent the views, positions, strategies or opinions of IBM or IBM management.

    technorati tags: , , , , ,,,,,,,,,,,,,

    Written by benchmarkingblog

    October 26, 2011 at 12:25 pm

    Whad’Ya Know – Great IBM Facts this Week

    with one comment

  • IBM Cloud revenue year to date has doubled full-year 2010 revenue.
  • There’s a new redpaper, Smarter Data Centers: Accelerating the Move to a Smarter Planet, with extremely “cool” case studies.
  • IBM announced the first Storage Performance Council (SPC) result with XIV, the very first on the SPC-2/E benchmark. The XIV Storage System demonstrated its ability to handle Big Data as well as providing associated energy use data. The SPC-2/E result showed that the XIV Storage System provides outstanding enterprise price-performance and Large File Processing (LFP) performance.(1)
  • IBM published a magnificent POWER7 BI TPC-H 3TB result on the IBM Power 780. This result was 3.9 times the performance per core and 19.7 times more storage efficient based on storage to database size than the best Oracle M9000 result. It was also 2.8 times less expensive.(2)
  • Newsweek named IBM Greenest Company In America.
  • And you can’t get much better than that.

    ************************************************

    (1)8259.94 MBPS SPC-2 (LFP) Data Rate, $137.07 SPC-2 (LFP) Price-Performance Source: Storage Performance Council SPC-2 Benchmark Results, http://www.storageperformance.org/results/benchmark_results_spc2, Results current as of 10/20/11.
    (2)IBM POWER 780 (8 sockets/32 cores/128 threads) 192,001 QphH@3000GB, $6.37/QphH@3000GB, available 11/30/11. Oracle M9000 (64 sockets/256 cores/256 threads) 386,478 QphH@3000GB, $18.19/QphH@3000GB, available 9/22/11.
    Source: http://www.tpc.org. Results current as of 10/20/11.

    SPC Benchmark-1 and SPC Benchmark-2 are trademarks of the Storage Performance Council.
    TPC-C ,TPC-H, and TPC-E are trademarks of the Transaction Performance Processing Council (TPPC).

    The postings on this site solely reflect the personal views of the author and do not necessarily represent the views, positions, strategies or opinions of IBM or IBM management.

    technorati tags: , , , , ,,,,,,,,,,,,,

    Written by benchmarkingblog

    October 20, 2011 at 5:27 pm

    Posted in SPC, TPC-H

    Tagged with , , ,

    Hey Oracle, TPC-H Shouldn’t be a Cakewalk

    with 3 comments

    Here in the northern hemisphere, tis the season of pumpkins and Fall Festivals. One of my favorite festival events just has to be the Cakewalk. Where can you play a game, win an amazing cake, and ensure a great dessert without even having to turn the oven on ?

    One of the ways that I’ve made sure to win one of the cakes is to buy a lot of tickets. If I get my friends to play on my behalf, I’ve really ensured that the probability that I’ll win is high. Disclosure: I do feel guilty when I do this and a little kid doesn’t win.

    And that’s what we are seeing with Oracle TPC-H results. Oracle’s latest BI benchmark result was announced last week on the new SPARC T4-4. Besides the high load time, unrealistically low maintenance coverage, and huge amount of storage needed, take a look at the queries run. Two facts stand out.

  • The query execution times are incredibly variable — for the same exact query. As an example, Query 6 had a maximum of 4140.2 — and a minimum of only 5.1 !
  • The number of streams of queries was 128. Oracle ran 128 times, IBM only 9. (1)
  • What does this mean? My colleague Juan asked me this just yesterday.

    TPC-H has a limited number of query variations; so when you run a lot of streams, you have a high probability that the same queries will be requested more than once. Oracle is greatly increasing the probability that they will have the results of the queries stored in their cache — which may not be representative of how their product would perform in a truly ad hoc query environment.

    Getting results this way, it’s a piece of cake.

    ************************************************

    (1)SPARC T4-4 server (4 sockets/32 cores/256 threads) 201,487 QphH@1000GB, $4.60/QphH@1000GB, available 10/30/11. IBM POWER 780 Model 9179-MHB server (8 sockets/32 cores/128 threads) 164,747.2 QphH@1000GB, $6.85/QphH@1000GB, available 3/31/11.
    Sources: http://www.spec.org, http://www.tpc.org. Results current as of 10/5/11.

    TPC-C ,TPC-H, and TPC-E are trademarks of the Transaction Performance Processing Council (TPPC).

    The postings on this site solely reflect the personal views of the author and do not necessarily represent the views, positions, strategies or opinions of IBM or IBM management.

    technorati tags: , , , , ,,,,,,,,,,,,,

    Written by benchmarkingblog

    October 5, 2011 at 11:59 am

    Posted in Oracle, SPARC T4, TPC-H

    Tagged with , , ,

    %d bloggers like this: