Elisabeth Stahl on Benchmarking and IT Optimization

All I Want for Christmas is a Good T4 Benchmark

with 7 comments

I read an article the other day about a Santa school where aspiring Santas go to learn about how to be Santa Claus. Important concepts like how to comb your beard, how to make that amazing Ho Ho Ho, and where to properly keep your hands when a kid is on your lap.

In this economy one of the new lessons that the Santa school offers is on how to reduce kids’ expectations. Santa’s workshop has too many toys to produce Johnny, Santa can’t fit everything into his sack, a bunch of elves got sick.

Oracle announced a new TPC-H BI 3TB benchmark result today on the Oracle SPARC T4-4 server. Expectations surely need to be reduced on this one because here’s what you need to know:

  • Oracle once again used an embarrassing amount of storage to run this benchmark — almost three times the amount of storage that IBM used.(1)
  • Oracle once again used 128 query streams for this benchmark compared with IBM’s 8. TPC-H has a limited number of query variations; so when you run a lot of streams, you have a high probability that the same queries will be requested more than once. Oracle is greatly increasing the probability that they will have the results of the queries stored in their cache — which may not be representative of how their product would perform in a truly ad hoc query environment.
  • Oracle once again published a benchmark where the configuration isn’t even available — until May 31, 2012.
  • Oracle once again included extremely minimal support in their pricing. Does $2300 a year sound like what you are paying for software “incident server support” . . . ?



The most important lesson for Santas — Never promise anything.


(1)IBM POWER 780 (8 sockets/32 cores/128 threads) 192,001 QphH@3000GB, $6.37/QphH@3000GB, available 11/30/11, 15,610 GB storage. Oracle SPARC T4-4 server (4 sockets/32 cores/256 threads) 205,792 QphH@3000GB, $4.10/QphH@3000GB, available 5/31/12, 45,600 GB storage.
Source: Results current as of 11/30/11.
TPC-C ,TPC-H, and TPC-E are trademarks of the Transaction Performance Processing Council (TPPC).

The postings on this site solely reflect the personal views of the author and do not necessarily represent the views, positions, strategies or opinions of IBM or IBM management.

technorati tags: , , , , ,,,,,,,,,,,,,


Written by benchmarkingblog

November 30, 2011 at 11:14 am

Posted in Oracle, POWER7, SPARC T4, TPC-H

Tagged with , , , , ,

7 Responses

Subscribe to comments with RSS.

  1. Sounds like you might be suggesting that Oracle is smarter at playing the benchmark game than IBM is?


    November 30, 2011 at 12:52 pm

  2. Kudos for a quick response! I don’t actually see the result on yet.

    Nearly all published benchmark results are tuned. It’s good someone calls attention to unusual configurations (like you’ve done), but the choice of storage or the number of query streams no more invalidates the result than would, say, running Sybase instead of DB2 on the 780.

    I don’t share your concerns over the points about availability date and pricing. Future availability dates are commonplace in these results. As long as these dates & pricing are permitted under the TPC-H run & reporting rules — and they appear to be — then your last two bullets seem more like objections to the benchmark itself than about Oracle’s result.

    Daniel Bowers

    November 30, 2011 at 1:58 pm

    • Dan, regarding pricing, I think it’s important to consider the audience for the Oracle offering – medium to large scale business with fairly intensive analytic needs. These business rarely purchase analytic solutions with rock bottom support (term licenses) that few customers would use in reality. It’s sort of gaming the system. Any company can put a support SKU on their price list that no one would buy for an enterprise deployment, then use it to calculate a ridiculously low $$/QphH. While it may meet the letter of the law for the benchmark publication terms, it’s more akin to a bait and switch that car dealers use to draw in customers to the lot and sell them something that they might actually drive home in. We’ve all seen those ads, but when you realize that the advertised car is a manual 4-speed with no radio, no air conditioning, and crank windows, you change your expectations or walk. I bet most customers will walk when the see the actual price, teaser pricing can be insulting for well educated and sophisticated IT buyers.

      Christian Reilly

      November 30, 2011 at 9:03 pm

      • Well the way I look at it, the SPARC T4 is smoking IBM’s Power7 and so IBM is trying really hard to nit pick irrelevant details like dates and claiming more storage and minimal support costs?.

        First let me summarize:
        The SPARC T4-4 (without use of F5100 this time)
        • 2x faster per CPU performance and 7% faster per system and per core than the IBM Power 780 server with eight POWER7 CPUs (total of 32 cores)
        • 36% better in price performance compared to the IBM Power 780 server even with IBM using Sybase
        • 29% faster than the IBM Power 780 for data loading.
        • Up to 3.4 times faster than the IBM Power 780 server for the Refresh Function.

        All this running Oracle DB versus IBM running Sybase! So who’s cheating now?

        If IBM wants to belly up to the bar, then submit a DB2 and AIX on Power7 result and show who’s the winner. Don’t have anything up your sleeves?

        So if you take out all the software and maintenance costs out of pricing analysis, now look how much worse off IBM is in the comparison! From the disclosures, the SPARC T4-4+Storage is 3x lower cost than Power7.


        December 1, 2011 at 9:14 am

  3. There are some staggering differences between hardware prices disclosed by IBM and Oracle with the TPC-H 1,000GB and 3,000GB benchmarks.

    TPC-H 1,000GB

    12/15/10 IBM Power 780
    Hardware & Maintenance List Price $1.5m USD / 164,747 QphH / $9.12 USD per QphH

    09/26/11Oracle SPARC T4-4
    Hardware & Maintenance List Price $0.75m USD / 201,847 QphH / $3.73 USD per QphH

    TPC-H 3,000GB

    10/14/11 IBM Power 780
    Hardware & Maintenance List Price $1.7m USD / 192,001 QphH / $9.05 USD per QphH

    11/30/11 Oracle SPARC T4-4
    Hardware & Maintenance List Price $0.45m USD / 205,792 QphH / $2.22 USD per QphH

    The hardware price performance of IBM Power 780 compared to Oracle SPARC T4-4 is 2.4x higher (TPC-H 1,000GB) or 4x higher (TPC-H 3,000GB).

    This data also appears to show the IBM Power 780’s hardware price performance ($ per QphH) has remained relatively static in almost 1 year. Yes, performance has increased, yet Power 780 customers are being asked to pay for these improvements.

    Conversely, in a little more than two months, Oracle’s SPARC T4 price performance would appear to have improved by greater than 40%.

    The database software differed for each vendor, with IBM opting for the very modestly priced Sybase IQ, with a similarly modest share of the enterprise database market.

    Let’s look at more sensible database software options for each vendor.

    IBM DB2 software licensing for the Power 780 would be 3,840 IBM PVUs. The SPARC T4-4 would require 3,200 PVUs, or 17% less than the Power 780.

    Oracle Database Enterprise Edition 11g r2 licenses for the Power 780 would amount to 64, since the Power 780 is physically a 64 core system. On the other hand, the SPARC T4-4 would only require 16 licenses, or a staggering 75% less than the Power 780.

    These benchmarks show emphatically divergent trends in performance and economics from IBM and Oracle.


    December 2, 2011 at 8:02 am

  4. I sure think it’s funny to comment on the 3 times the storage. I would love to have 3 times the storage and the servers all for a lower price. Sign me up!!


    February 16, 2012 at 10:58 am

  5. So why we rnning sybase instead of using DB2?


    August 15, 2012 at 7:58 am

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: