benchmarkingblog

Elisabeth Stahl on Benchmarking and IT Optimization

Hey Oracle, TPC-H Shouldn’t be a Cakewalk

with 3 comments

Here in the northern hemisphere, tis the season of pumpkins and Fall Festivals. One of my favorite festival events just has to be the Cakewalk. Where can you play a game, win an amazing cake, and ensure a great dessert without even having to turn the oven on ?

One of the ways that I’ve made sure to win one of the cakes is to buy a lot of tickets. If I get my friends to play on my behalf, I’ve really ensured that the probability that I’ll win is high. Disclosure: I do feel guilty when I do this and a little kid doesn’t win.

And that’s what we are seeing with Oracle TPC-H results. Oracle’s latest BI benchmark result was announced last week on the new SPARC T4-4. Besides the high load time, unrealistically low maintenance coverage, and huge amount of storage needed, take a look at the queries run. Two facts stand out.

  • The query execution times are incredibly variable — for the same exact query. As an example, Query 6 had a maximum of 4140.2 — and a minimum of only 5.1 !
  • The number of streams of queries was 128. Oracle ran 128 times, IBM only 9. (1)
  • What does this mean? My colleague Juan asked me this just yesterday.

    TPC-H has a limited number of query variations; so when you run a lot of streams, you have a high probability that the same queries will be requested more than once. Oracle is greatly increasing the probability that they will have the results of the queries stored in their cache — which may not be representative of how their product would perform in a truly ad hoc query environment.

    Getting results this way, it’s a piece of cake.

    ************************************************

    (1)SPARC T4-4 server (4 sockets/32 cores/256 threads) 201,487 QphH@1000GB, $4.60/QphH@1000GB, available 10/30/11. IBM POWER 780 Model 9179-MHB server (8 sockets/32 cores/128 threads) 164,747.2 QphH@1000GB, $6.85/QphH@1000GB, available 3/31/11.
    Sources: http://www.spec.org, http://www.tpc.org. Results current as of 10/5/11.

    TPC-C ,TPC-H, and TPC-E are trademarks of the Transaction Performance Processing Council (TPPC).

    The postings on this site solely reflect the personal views of the author and do not necessarily represent the views, positions, strategies or opinions of IBM or IBM management.

    technorati tags: , , , , ,,,,,,,,,,,,,

    Advertisements

    Written by benchmarkingblog

    October 5, 2011 at 11:59 am

    Posted in Oracle, SPARC T4, TPC-H

    Tagged with , , ,

    3 Responses

    Subscribe to comments with RSS.

    1. Hello, Elisabeth.
      Can you tell how can we conduct our own TPC-H benchmark?

      Alex

      November 15, 2012 at 5:43 am


    Leave a Reply

    Fill in your details below or click an icon to log in:

    WordPress.com Logo

    You are commenting using your WordPress.com account. Log Out / Change )

    Twitter picture

    You are commenting using your Twitter account. Log Out / Change )

    Facebook photo

    You are commenting using your Facebook account. Log Out / Change )

    Google+ photo

    You are commenting using your Google+ account. Log Out / Change )

    Connecting to %s

    %d bloggers like this: