benchmarkingblog

Benchmarking and Systems Performance

SPARC T4 to the core

with 5 comments

Yesterday I went apple picking in rural Ohio. That makes sense.

It’s not something that most people associate with California even though California is actually one of the top apple-producing states. But it works rather well for this SPARC analysis.

I usually love apple picking – with the doomed sun of autumn, the crunchy sweetness of the fruit, the dog wolfing down the cores. But there were certain aspects of my trip yesterday that were plainly unimpressive.

Sort of like the latest SPARC T4 benchmark results announced by Oracle today:

  • Oracle claimed nine T4 world records. 7 of the 9 are not industry standard benchmarks but Oracle’s own benchmarks, most based on internal testing. Sort of like when we called the orchard and they said that many varieties were available for picking. When we got there, only a few could really be picked. Where was that renowned low hanging fruit?
  • Some Oracle claims compared the new T4 results with previous benchmark versions, never a good idea. Like encouraging your kids to climb on the fruit-bearing trees. Some results compared Oracle to Oracle. If you read carefully, some didn’t compare to anything.
  • Oracle claimed a “generational increase in performance” over previous versions. Note that this claim (which has no published benchmarks behind it) focuses on single threaded applications – how many of those do you have? And you can easily get a 5x improvement when you start from a very very small seed.
  • Oracle’s SPECjEnterprise2010 Java T4 benchmark result, which was highlighted, needed four times the number of app nodes, twice the number of cores, almost four times the amount of memory and significantly more storage than the IBM POWER7 result.(1) Oracle’s price performance and space metric claims (which are not even official benchmark metrics) were calculated only for the application tier of this benchmark, basically ignoring the all important database server, software and storage. Sort of like eating only the pulp of the apple and ignoring all the vitamins in the skin.
  • Oracle’s T4 TPC-H 1TB BI benchmark result, another benchmark which was highlighted, actually had a longer load time than the IBM result from last year. Oracle’s storage use was ludicrous, like the number of apples my Labrador ended up eating; Oracle’s total storage needed to the database size ratio was 10.80 compared to the IBM value of 3.97. Oracle needed 128 streams of queries, IBM only 9. And make sure to note the extremely low and unrealistic Oracle maintenance costs used to get to the price performance number.(2)
  • The range and results of these benchmarks are ultimately disappointing. Instead of making a wonderful pie and apple rings last night, we swept up chips of dried orchard mud in the dark.

    ************************************************

    (1)Oracle WebLogic Server 11g and Oracle Database 11g Release 2 with Oracle Real Application Clusters and Oracle Solaris running on a four-node SPARC T4-4 cluster, each system with four SPARC T4 3GHz processors, (128 core app server, 64 core db server), 40,104.86 SPECjEnterprise2010 EjOPS vs. WebSphere Application Server V7 on IBM Power 780 and DB2 on IBM Power 750 Express, (64 core app server, 32 core db server), 16,646.34 SPECjEnterprise2010 EjOPS.
    (2)SPARC T4-4 server (4 sockets/32 cores/256 threads) 201,487 QphH@1000GB, $4.60/QphH@1000GB, available 10/30/11. IBM POWER 780 Model 9179-MHB server (8 sockets/32 cores/128 threads) 164,747.2 QphH@1000GB, $6.85/QphH@1000GB, available 3/31/11.
    Sources: http://www.spec.org, http://www.tpc.org. Results current as of 9/26/11.

    SPEC, SPECint, SPECfp, SPECjbb, SPECweb, SPECjAppServer, SPECjEnterprise, SPECjvm, SPECvirt, SPECompM, SPECompL, SPECsfs, SPECpower, SPEC MPI and SPECpower_ssj are trademarks of the Standard Performance Evaluation Corporation (SPEC).

    TPC-C ,TPC-H, and TPC-E are trademarks of the Transaction Performance Processing Council (TPPC).

    The postings on this site solely reflect the personal views of the author and do not necessarily represent the views, positions, strategies or opinions of IBM or IBM management.

    technorati tags: , , , , ,,,,,,,,,,,,,

    About these ads

    Written by benchmarkingblog

    September 26, 2011 at 9:43 pm

    5 Responses

    Subscribe to comments with RSS.

    1. Oracle was very quick to point out alot of benchmarks EXCEPT the SPEC_INT. I got to thinking about that and realized that Right now the Oracle T3-4 system has a SPEC_INT_RATE of 666. If we are to believe that the T4 is a 5x speed increase, then the SPEC_INT_RATE for the new T4-4 will logically be ~3300. What are the marketing gurus and Larry Ellison going to say when the SPEC_INT_RATE comes in much lower than that?

      5x speed jump? um, yeah. I’m not buying it.

      jeff bethke (@jeffbethke)

      September 27, 2011 at 3:57 pm

      • SPEC CPU2006 is a useless benchmark for evaluating real world workload performance so I doubt Oracle will be publishing for SPARC T4. SPARC T4 is for running real world enterprise workloads real fast. Sure its great to test how fast a CPU is when running the benchmark in cache, and even testing memory bandwidth, but as the SPEC CPU FAQS state here: http://www.spec.org/cpu2006/Docs/readme1st.html, “SPEC CPU2006 is not intended to stress other computer components such as networking, the operating system, graphics, or the I/O system” So for example, tell me one real world application environment, say a database, or Java or application server, where the OS isnt used, networking isnt involved, and most importantly, I/O isn’t tested? SPEC CPU misleads everyone into thinking whos fastest. Its kind of like putting a car with a very powerful engine on a dyno test, only to realize when you put it on the road, the skinny tires blow up above 100 mph!

        Phil

        October 3, 2011 at 5:06 am

        • I was involved with benchmarking Sun Microsystems servers in the late 80s and early 90s when they had blistering performance all round and they published results for every common benchmark known to man (and won). Now that their CPUs are not so fast they no longer publish industry standard benchmark results. SPARC T4 is a new CPU so it make perfect sense to benchmark the CPU!!

          Please do not claim that SPEC CPU2006 is a useless benchmark….I am an infrastructure architect at a very large site and will never willingly purchase servers that do not provide basic information like CPU speed. Besides, last time I ran my own benchmark CPU suite on Oracle TT340 it was soundly beaten my tiny 2011 Macbook AIr. If an application is CPU bound, you need fast CPU….no matter what Oracle argue!

          This Is Private

          October 6, 2011 at 12:08 am

        • I guess Phil is just happy to buy a sports car from a manufacturer that won’t reveal metrics such as top speed, acceleration, power or weight but rather trust the claim that the car is really, really fast on the manufacturers own race track.

          Car analogy gone bad

          October 6, 2011 at 8:51 am

    2. I also disagree with both Phil and Jeff.

      Regarding Jeff, I believe the 5x speedup claim was referring to single thread specint performance. That’s not as great a number as one should have hoped for. Theoretically, with clock speed 2x that of T3, and with 1 thread instead of 8 doing the work, 16x would have been the speedup. But I’ll take 5 as it is a great improvement. Note that the chip has half the cores and 2x the clock speed, so I’d expect CPU2006 _Rate_ benchmarks to be similar to that of the T3. Which is still pretty good. The whole point of the T4 is single thread performance.

      Real world performance is NOT just about total throughput. The CPU2006 benchmarks are useful, because there is very little you can do the tweak it, except design your chips a certain way. I find TCP benchmarks more dubious, and pretty useless for measuring a chip’s performance, as it is too dependent on the disk system, or even implementation of the benchmark itself. It takes a lot of effort and tuning of the benchmark itself to get good TCP-C or TCP-H result, so the number you read is more a reflection of the benchmarking team than the hardware itself. I tend to ignore such benchmarks when it comes to planning CPU capacity., I also ignore spec CPU2006 peak results, and only use the base, as it is rarely the case that every single program in a real life deployment will be tuned for optimal compiler parameters.

      There will always be some sequential operations that ultimately limit your scalability. And for many applications the slow single thread performance made Coolthreads an unfit platform, even for database servers. Most applications out there are legacy applications, and cannot easily be written to take advantage of massive parallelism. It all depends on the application. IBM’s power series have dominated for a long time over Sun, as Sun’s single thread performance was standing still or going backwards. With the T4, Sun at least becomes a viable alternative again. It doesn’t need to break any records, it just needs to be viable. I’ve been waiting for many years for Sun to see the light, and get away from their parallell-only mindset. The real world needs chips that are fast and big, not just big, or just fast.

      Bjorn

      October 21, 2011 at 10:59 am


    Leave a Reply

    Fill in your details below or click an icon to log in:

    WordPress.com Logo

    You are commenting using your WordPress.com account. Log Out / Change )

    Twitter picture

    You are commenting using your Twitter account. Log Out / Change )

    Facebook photo

    You are commenting using your Facebook account. Log Out / Change )

    Google+ photo

    You are commenting using your Google+ account. Log Out / Change )

    Connecting to %s

    %d bloggers like this: