benchmarkingblog

Elisabeth Stahl on Benchmarking and IT Optimization

Posts Tagged ‘TPC-H

Come on Oracle, Get “With It” Benchmarking

leave a comment »

I admit that many weekends this time of year you will find me (when I’m not enslaved by the leaf blower) curled up with a good old book on the old couch with my thankfully not so old dog.

But this weekend I truly was “with it.”

On Saturday night I attended one of the most sought-after sold out concert events ever to hit this town. I got to see a Pink concert that included not only 17 of her best songs but Pink flying through the air doing acrobatics that you simply would not believe. A rock concert rolled right into the circus, truly amazing.

And then to top it all off, on Sunday night I attended one of the most sought-after sold out movie events of the year. I got to see the latest Hunger Games, Catching Fire, second in this awesome trilogy which could be even more popular than the first.

So on Monday when I saw the latest Oracle SPARC T5-4 benchmark result on the TPC-H decision support benchmark (1), all I could think was how so “not with it.”

Like Gangnam Style this year. Or What Does the Fox Say this month.

Hey, I’m the first one to like legacy. My closet is filled with vintage looks. I love retro — just not when it comes to benchmarks.

Here is what you need to know.

  • First of all, this is TPC-H. Yawn. We’re ready for something new here.
  • Most of the TPC-H results are grayed out in this category, considered “historic.” This result is right next to a result from IBM — from 2007 (yes you heard that right).
  • Total Storage to Database Size ratio is a massive 60.80. Talk about overkill on storage to achieve performance. This number is many many times the ratio we’ve seen from other results.
  • Load time is a whopping 9.63 hours.
  • 128 query streams are needed. Most results use many, many fewer. That’s because TPC-H has a limited number of query variations; so when you run a lot of streams, you have a high probability that the same queries will be requested more than once. Oracle is greatly increasing the probability that they will have the results of the queries stored in their cache — which may not be representative of how their product would perform in a truly ad hoc query environment.
  • Oracle once again included extremely minimal support in their pricing. Does $2300 a year sound like what you are paying for software support?

************************************************
(1) Oracle TPC-H of 377,594 QphH@10000GB,$4.65 per QphH,Availability 11/25/13,Oracle Database 11g R2 Enterprise Edition w/Partitioning,SPARC T5 3.6 GHz; Total # of Processors: 4,Total # of Cores: 64,Total # of Threads: 512.
Source: http://www.tpc.org. Results current as of 11/25/13.

TPC-C ,TPC-H, and TPC-E are trademarks of the Transaction Performance Processing Council (TPPC).

The postings on this site solely reflect the personal views of the author and do not necessarily represent the views, positions, strategies or opinions of IBM or IBM management.

technorati tags: , , ,,,,,,,,,,,,,

,,

Advertisements

Written by benchmarkingblog

November 26, 2013 at 11:28 am

Posted in SPARC T5, TPC-H

Tagged with , ,

The National Security on the T5-4 and Big Data

with 5 comments

There’s been a lot of talk the last few days on Big Data and when it’s “right” to capture and use it. Some say it’s a real invasion of privacy. Others realistically point out that it is the best way to counter terrorism.

Whichever you believe, the important thing is that Big Data is being discussed not just in geeky meetings with IT managers but by everybody. When your neighbor across the street stops trimming his tree branches just to talk to you about it, you know it’s hot stuff.

So I was particularly interested to see that Oracle just published a new TPC-H data benchmark result on the SPARC T5-4.

And here is what hits you like a train.

  • Why is this publish at only the 3TB size when all the talk these days is on much larger amounts of data?
  • Why is the Total Storage to Database Size ratio a whopping 29? Talk about overkill on storage to achieve performance. This number is many times the ratio we’ve seen from other results.
  • Why is the memory to database size % a whopping 66.6? Again, much more than you should need and what we normally see.
  • Why are there 192 query streams needed? Most results use many, many fewer. That’s because TPC-H has a limited number of query variations; so when you run a lot of streams, you have a high probability that the same queries will be requested more than once. Oracle is greatly increasing the probability that they will have the results of the queries stored in their cache — which may not be representative of how their product would perform in a truly ad hoc query environment.
  • Why isn’t the configuration available now? Because key elements of the storage are not ready.
  • Why did Oracle once again include extremely minimal support in their pricing? Does $2300 a year sound like what you are paying for software “incident server support” . . . ? You don’t even need to answer this one.

Comments are welcome at your own risk.

************************************************
(1) Oracle TPC-H of 409,721 QphH@3000GB,$3.94 per QphH,Availability 09/24/13,Oracle Database 11g R2 Enterprise Edition w/Partitioning,SPARC T5 3.6 GHz; Total # of Processors: 4,Total # of Cores: 64,Total # of Threads: 512.
Source: http://www.tpc.org. Results current as of 6/12/13.
TPC-C ,TPC-H, and TPC-E are trademarks of the Transaction Performance Processing Council (TPPC).

The postings on this site solely reflect the personal views of the author and do not necessarily represent the views, positions, strategies or opinions of IBM or IBM management.

technorati tags: , , ,,,,,,,,,,,,,,

,

Written by benchmarkingblog

June 12, 2013 at 3:36 pm

Posted in SPARC T5, TPC-H

Tagged with , , ,

I Want to Buy a Zoo, Not an Oracle System

with 5 comments

Most people don’t think about going to the zoo in the middle of winter. But it’s actually the very best time to go.

No crowds, the zebra fried oreo shacks are closed, and the animals are at their very best. And, after all, isn’t that what it’s really all about?

Last week, I had a spectacular visit. The polar bears were playing with dead Christmas trees, the seals and sea lions were playing with their rubber balls, and the grizzly cubs were playing with themselves.

And then there’s the Rainforest. Imagine stepping from a cold snowy Cleveland day into a zoological tropical paradise. Sort of like the feeling you get when the plane doors open, you get your carry-on from the overhead, and you step off in Miami. My rainforest favorites are the tropical monkeys, the river otters, and, of course, the anteaters. But alas, last week, the anteaters were still, lying on their sides, hiding behind a crop of rocks.

And that’s how I’ve been feeling lately about Oracle.

  • See what’s hiding behind the Oracle SPARC SuperCluster and Exadata systems. “Must buy” storage server software.
  • See what’s hiding behind that pricing in the Oracle benchmark. Artificially low support costs.
  • See what’s hiding behind Oracle’s “Itanium roadmap.” An investigation into Oracle’s “potentially abusive” practices.
  • What else do YOU think Oracle is hiding? All thoughts welcome.

    ************************************************

    Oracle SPARC T4-4 server (4 sockets/32 cores/256 threads) 205,792 QphH@3000GB, $4.10/QphH@3000GB, available 5/31/12.
    Source: http://www.tpc.org. Results current as of 1/11/12.
    TPC-C ,TPC-H, and TPC-E are trademarks of the Transaction Performance Processing Council (TPPC).

    The postings on this site solely reflect the personal views of the author and do not necessarily represent the views, positions, strategies or opinions of IBM or IBM management.

    technorati tags: , , , , ,,,,,,,,,,,,,

    Written by benchmarkingblog

    January 11, 2012 at 1:41 pm

    All I Want for Christmas is a Good T4 Benchmark

    with 7 comments

    I read an article the other day about a Santa school where aspiring Santas go to learn about how to be Santa Claus. Important concepts like how to comb your beard, how to make that amazing Ho Ho Ho, and where to properly keep your hands when a kid is on your lap.

    In this economy one of the new lessons that the Santa school offers is on how to reduce kids’ expectations. Santa’s workshop has too many toys to produce Johnny, Santa can’t fit everything into his sack, a bunch of elves got sick.

    Oracle announced a new TPC-H BI 3TB benchmark result today on the Oracle SPARC T4-4 server. Expectations surely need to be reduced on this one because here’s what you need to know:

    • Oracle once again used an embarrassing amount of storage to run this benchmark — almost three times the amount of storage that IBM used.(1)
    • Oracle once again used 128 query streams for this benchmark compared with IBM’s 8. TPC-H has a limited number of query variations; so when you run a lot of streams, you have a high probability that the same queries will be requested more than once. Oracle is greatly increasing the probability that they will have the results of the queries stored in their cache — which may not be representative of how their product would perform in a truly ad hoc query environment.
    • Oracle once again published a benchmark where the configuration isn’t even available — until May 31, 2012.
    • Oracle once again included extremely minimal support in their pricing. Does $2300 a year sound like what you are paying for software “incident server support” . . . ?

     

     

    The most important lesson for Santas — Never promise anything.

    ************************************************

    (1)IBM POWER 780 (8 sockets/32 cores/128 threads) 192,001 QphH@3000GB, $6.37/QphH@3000GB, available 11/30/11, 15,610 GB storage. Oracle SPARC T4-4 server (4 sockets/32 cores/256 threads) 205,792 QphH@3000GB, $4.10/QphH@3000GB, available 5/31/12, 45,600 GB storage.
    Source: http://www.tpc.org. Results current as of 11/30/11.
    TPC-C ,TPC-H, and TPC-E are trademarks of the Transaction Performance Processing Council (TPPC).

    The postings on this site solely reflect the personal views of the author and do not necessarily represent the views, positions, strategies or opinions of IBM or IBM management.

    technorati tags: , , , , ,,,,,,,,,,,,,

    Written by benchmarkingblog

    November 30, 2011 at 11:14 am

    Posted in Oracle, POWER7, SPARC T4, TPC-H

    Tagged with , , , , ,

    Whad’Ya Know – Great IBM Facts this Week

    with one comment

  • IBM Cloud revenue year to date has doubled full-year 2010 revenue.
  • There’s a new redpaper, Smarter Data Centers: Accelerating the Move to a Smarter Planet, with extremely “cool” case studies.
  • IBM announced the first Storage Performance Council (SPC) result with XIV, the very first on the SPC-2/E benchmark. The XIV Storage System demonstrated its ability to handle Big Data as well as providing associated energy use data. The SPC-2/E result showed that the XIV Storage System provides outstanding enterprise price-performance and Large File Processing (LFP) performance.(1)
  • IBM published a magnificent POWER7 BI TPC-H 3TB result on the IBM Power 780. This result was 3.9 times the performance per core and 19.7 times more storage efficient based on storage to database size than the best Oracle M9000 result. It was also 2.8 times less expensive.(2)
  • Newsweek named IBM Greenest Company In America.
  • And you can’t get much better than that.

    ************************************************

    (1)8259.94 MBPS SPC-2 (LFP) Data Rate, $137.07 SPC-2 (LFP) Price-Performance Source: Storage Performance Council SPC-2 Benchmark Results, http://www.storageperformance.org/results/benchmark_results_spc2, Results current as of 10/20/11.
    (2)IBM POWER 780 (8 sockets/32 cores/128 threads) 192,001 QphH@3000GB, $6.37/QphH@3000GB, available 11/30/11. Oracle M9000 (64 sockets/256 cores/256 threads) 386,478 QphH@3000GB, $18.19/QphH@3000GB, available 9/22/11.
    Source: http://www.tpc.org. Results current as of 10/20/11.

    SPC Benchmark-1 and SPC Benchmark-2 are trademarks of the Storage Performance Council.
    TPC-C ,TPC-H, and TPC-E are trademarks of the Transaction Performance Processing Council (TPPC).

    The postings on this site solely reflect the personal views of the author and do not necessarily represent the views, positions, strategies or opinions of IBM or IBM management.

    technorati tags: , , , , ,,,,,,,,,,,,,

    Written by benchmarkingblog

    October 20, 2011 at 5:27 pm

    Posted in SPC, TPC-H

    Tagged with , , ,

    Hey Oracle, TPC-H Shouldn’t be a Cakewalk

    with 3 comments

    Here in the northern hemisphere, tis the season of pumpkins and Fall Festivals. One of my favorite festival events just has to be the Cakewalk. Where can you play a game, win an amazing cake, and ensure a great dessert without even having to turn the oven on ?

    One of the ways that I’ve made sure to win one of the cakes is to buy a lot of tickets. If I get my friends to play on my behalf, I’ve really ensured that the probability that I’ll win is high. Disclosure: I do feel guilty when I do this and a little kid doesn’t win.

    And that’s what we are seeing with Oracle TPC-H results. Oracle’s latest BI benchmark result was announced last week on the new SPARC T4-4. Besides the high load time, unrealistically low maintenance coverage, and huge amount of storage needed, take a look at the queries run. Two facts stand out.

  • The query execution times are incredibly variable — for the same exact query. As an example, Query 6 had a maximum of 4140.2 — and a minimum of only 5.1 !
  • The number of streams of queries was 128. Oracle ran 128 times, IBM only 9. (1)
  • What does this mean? My colleague Juan asked me this just yesterday.

    TPC-H has a limited number of query variations; so when you run a lot of streams, you have a high probability that the same queries will be requested more than once. Oracle is greatly increasing the probability that they will have the results of the queries stored in their cache — which may not be representative of how their product would perform in a truly ad hoc query environment.

    Getting results this way, it’s a piece of cake.

    ************************************************

    (1)SPARC T4-4 server (4 sockets/32 cores/256 threads) 201,487 QphH@1000GB, $4.60/QphH@1000GB, available 10/30/11. IBM POWER 780 Model 9179-MHB server (8 sockets/32 cores/128 threads) 164,747.2 QphH@1000GB, $6.85/QphH@1000GB, available 3/31/11.
    Sources: http://www.spec.org, http://www.tpc.org. Results current as of 10/5/11.

    TPC-C ,TPC-H, and TPC-E are trademarks of the Transaction Performance Processing Council (TPPC).

    The postings on this site solely reflect the personal views of the author and do not necessarily represent the views, positions, strategies or opinions of IBM or IBM management.

    technorati tags: , , , , ,,,,,,,,,,,,,

    Written by benchmarkingblog

    October 5, 2011 at 11:59 am

    Posted in Oracle, SPARC T4, TPC-H

    Tagged with , , ,

    SPARC T4 to the core

    with 5 comments

    Yesterday I went apple picking in rural Ohio. That makes sense.

    It’s not something that most people associate with California even though California is actually one of the top apple-producing states. But it works rather well for this SPARC analysis.

    I usually love apple picking – with the doomed sun of autumn, the crunchy sweetness of the fruit, the dog wolfing down the cores. But there were certain aspects of my trip yesterday that were plainly unimpressive.

    Sort of like the latest SPARC T4 benchmark results announced by Oracle today:

  • Oracle claimed nine T4 world records. 7 of the 9 are not industry standard benchmarks but Oracle’s own benchmarks, most based on internal testing. Sort of like when we called the orchard and they said that many varieties were available for picking. When we got there, only a few could really be picked. Where was that renowned low hanging fruit?
  • Some Oracle claims compared the new T4 results with previous benchmark versions, never a good idea. Like encouraging your kids to climb on the fruit-bearing trees. Some results compared Oracle to Oracle. If you read carefully, some didn’t compare to anything.
  • Oracle claimed a “generational increase in performance” over previous versions. Note that this claim (which has no published benchmarks behind it) focuses on single threaded applications – how many of those do you have? And you can easily get a 5x improvement when you start from a very very small seed.
  • Oracle’s SPECjEnterprise2010 Java T4 benchmark result, which was highlighted, needed four times the number of app nodes, twice the number of cores, almost four times the amount of memory and significantly more storage than the IBM POWER7 result.(1) Oracle’s price performance and space metric claims (which are not even official benchmark metrics) were calculated only for the application tier of this benchmark, basically ignoring the all important database server, software and storage. Sort of like eating only the pulp of the apple and ignoring all the vitamins in the skin.
  • Oracle’s T4 TPC-H 1TB BI benchmark result, another benchmark which was highlighted, actually had a longer load time than the IBM result from last year. Oracle’s storage use was ludicrous, like the number of apples my Labrador ended up eating; Oracle’s total storage needed to the database size ratio was 10.80 compared to the IBM value of 3.97. Oracle needed 128 streams of queries, IBM only 9. And make sure to note the extremely low and unrealistic Oracle maintenance costs used to get to the price performance number.(2)
  • The range and results of these benchmarks are ultimately disappointing. Instead of making a wonderful pie and apple rings last night, we swept up chips of dried orchard mud in the dark.

    ************************************************

    (1)Oracle WebLogic Server 11g and Oracle Database 11g Release 2 with Oracle Real Application Clusters and Oracle Solaris running on a four-node SPARC T4-4 cluster, each system with four SPARC T4 3GHz processors, (128 core app server, 64 core db server), 40,104.86 SPECjEnterprise2010 EjOPS vs. WebSphere Application Server V7 on IBM Power 780 and DB2 on IBM Power 750 Express, (64 core app server, 32 core db server), 16,646.34 SPECjEnterprise2010 EjOPS.
    (2)SPARC T4-4 server (4 sockets/32 cores/256 threads) 201,487 QphH@1000GB, $4.60/QphH@1000GB, available 10/30/11. IBM POWER 780 Model 9179-MHB server (8 sockets/32 cores/128 threads) 164,747.2 QphH@1000GB, $6.85/QphH@1000GB, available 3/31/11.
    Sources: http://www.spec.org, http://www.tpc.org. Results current as of 9/26/11.

    SPEC, SPECint, SPECfp, SPECjbb, SPECweb, SPECjAppServer, SPECjEnterprise, SPECjvm, SPECvirt, SPECompM, SPECompL, SPECsfs, SPECpower, SPEC MPI and SPECpower_ssj are trademarks of the Standard Performance Evaluation Corporation (SPEC).

    TPC-C ,TPC-H, and TPC-E are trademarks of the Transaction Performance Processing Council (TPPC).

    The postings on this site solely reflect the personal views of the author and do not necessarily represent the views, positions, strategies or opinions of IBM or IBM management.

    technorati tags: , , , , ,,,,,,,,,,,,,

    Written by benchmarkingblog

    September 26, 2011 at 9:43 pm

    %d bloggers like this: