Elisabeth Stahl on Benchmarking and IT Optimization

Is There a “Right” and “Wrong” with Benchmarks?

with 4 comments

Like many of our politicians lately, Oracle has been proselytizing — in this case that there are certain benchmarks which are right and certain benchmarks which are wrong.

In their evaluation Oracle claims to focus on all the right benchmarks while claiming that everyone else is focusing on the wrong ones. Let’s take a look at their reviews:

  • Thumbs up for SPECjEnterprise – Yes, I wholeheartedly agree, a good benchmark to use for Java applications. And that’s why IBM has outstanding results on this benchmark. Oracle’s T4 result needed four times the number of app nodes, twice the number of cores, almost four times the amount of memory and significantly more storage than the IBM POWER7 result.(1)
  • Thumbs up for Oracle ebusiness, JDEdwards, PeopleSoft, Siebel – Good when you need to model these exact applications but keep in mind that Oracle owns these “independent” benchmarks. Even with a stacked deck, IBM has #1’s here.
  • Thumbs up for TPC-H – Yes, totally agree that this benchmark is valuable for business intelligence applications. Which is why IBM just published a leadership 3TB TPC-H result.(2) And Oracle’s T4 result needed 2.7x the amount of storage and 14x the number of streams than the IBM POWER7 1TB result.(3)
  • Thumbs up for SPECweb2005 – Funny that Oracle likes this one since there is a definite lack of publishes here anymore and this benchmark is being retired in January.
  • Thumbs up for TPC-C – Even funnier that Oracle likes this one since Sun avoided publishing on this for many, many years claiming this benchmark was too old. Now, as a fair-weather friend, it’s a favorite. IBM has the top nonclustered result (even with an older Power system).(4)
  • Thumbs down for SAP SD – Interesting that this valuable benchmark for SAP users is on Oracle’s hit list. This benchmark is a wonderful measure for SAP users and has many proof points from many vendors. Is that why Oracle tries to avoid this one and instead publishes on other more obscure SAP benchmarks like SD-Parallel and ATO?
  • Thumbs down for SPECjAppServer2004 – Looking to fill up space, since Oracle knows as well as we do that this benchmark was retired almost a year ago. Also ironic because I do seem to remember that Oracle/Sun used to really like this benchmark. Alot.
  • Thumbs down for SPECcpu and SPECjbb – These are nice and easy for integer, floating point, and Java business apps. Surely valuable in their own way.
  • Thumbs down for Stream – Ignoring HPC users are we? We’ve seen this before with the TOP500. IBM has the most systems with 213. Oracle had 12.
  • Keep in mind that Oracle has only published their latest SPARC T4 results on 2 industry standard benchmarks — that’s right, only 2. Which is fine, if you don’t mind steering with your eyes closed.

    My view is that you may want and need to look at different aspects of systems across a wide portfolio of many different types of benchmarks. Which is why IBM is #1 — in over 100.


    (1)Oracle WebLogic Server 11g and Oracle Database 11g Release 2 with Oracle Real Application Clusters and Oracle Solaris running on a four-node SPARC T4-4 cluster, each system with four SPARC T4 3GHz processors, (128 core app server, 64 core db server), 40,104.86 SPECjEnterprise2010 EjOPS vs. WebSphere Application Server V7 on IBM Power 780 and DB2 on IBM Power 750 Express, (64 core app server, 32 core db server), 16,646.34 SPECjEnterprise2010 EjOPS.
    (2)IBM POWER 780 (8 sockets/32 cores/128 threads) 192,001 QphH@3000GB, $6.37/QphH@3000GB, available 11/30/11.
    (3)SPARC T4-4 server (4 sockets/32 cores/256 threads) 201,487 QphH@1000GB, $4.60/QphH@1000GB, available 10/30/11. IBM POWER 780 Model 9179-MHB server (8 sockets/32 cores/128 threads) 164,747.2 QphH@1000GB, $6.85/QphH@1000GB, available 3/31/11.
    (4)A 64-core IBM Power 595 (5 GHz, 32 chips, 128 threads) with IBM DB2 9.5 is the best overall single system (6,085,166 tpmC, $2.81/tpmC, configuration available 12/10/08).
    Sources:,, Results current as of 10/26/11.

    TPC-C ,TPC-H, and TPC-E are trademarks of the Transaction Performance Processing Council (TPPC).

    SAP, mySAP and other SAP product and service names mentioned herein as well as their respective
    logos are trademarks or registered trademarks of SAP AG in Germany and in several other countries all
    over the world.

    SPEC, SPECint, SPECfp, SPECjbb, SPECweb, SPECjAppServer, SPECjEnterprise, SPECjvm, SPECvirt, SPECompM, SPECompL, SPECsfs, SPECpower, SPEC MPI and SPECpower_ssj are trademarks of the Standard Performance Evaluation Corporation (SPEC).

    The postings on this site solely reflect the personal views of the author and do not necessarily represent the views, positions, strategies or opinions of IBM or IBM management.

    technorati tags: , , , , ,,,,,,,,,,,,,


    Written by benchmarkingblog

    October 26, 2011 at 12:25 pm

    4 Responses

    Subscribe to comments with RSS.

    1. Is there a reference to this list of Oracle poo-pooed benchmarks on Oracle’s website or other? I’d love to see that.


      November 10, 2011 at 11:41 am

    2. “you may want and need to look at different aspects of systems across a wide portfolio of many different types of benchmarks”

      If you do that no way in hell you would choose IBM…

      Both AIX and Linux are quite behind Solaris functionality wise. (nothing comparable to Dtrace and ZFS)

      If you look at DB2 for Linux, you get something equivalent with Oracle 7. (multi version concurrency control only introduced in 2009, you kidding me? Oracle had this since version 3, and always had it on the Linux version).

      Oh, almoust formot to mention Websphere, buggy, slow, bloated compared to Weblogic.
      And the famous IBM MQ that garbles messages when compresison is enabled…. Do you guys do any QA on your products?

      Funny enough if IBM would have managed to buy Sun as they intended you would have singed a different song…

      Z R

      November 29, 2011 at 4:57 pm

      • Hey ZR – Thanks for your comments. But I think you are definitely confusing performance with other functions. You should submit any errors to software support. Thanks !


        November 30, 2011 at 9:47 am

      • There are no multi version concurrency control in db2 …
        last_committed in db2 can’t provide consistent snapshot.


        December 7, 2011 at 5:43 pm

    Leave a Reply

    Fill in your details below or click an icon to log in: Logo

    You are commenting using your account. Log Out /  Change )

    Google+ photo

    You are commenting using your Google+ account. Log Out /  Change )

    Twitter picture

    You are commenting using your Twitter account. Log Out /  Change )

    Facebook photo

    You are commenting using your Facebook account. Log Out /  Change )


    Connecting to %s

    %d bloggers like this: