benchmarkingblog

Benchmarking and Systems Performance

Shoe Fetish or Benchmark Comparison ?

with 5 comments

Last month I visited the Fashion Institute of Technology’s new exhibit “Shoe Obsession.” And for anyone who relishes shoes, this was the place to be. You enter the dark rooms and the glass cases are absolutely glowing in light, highlighting the SHOES. There’s Manolo Blahnik, Christian Louboutin, Prada and many more, as far as the eye can see. Each shoe is made out of a huge array of materials — plastics, metals, beads, ribbons, velvet, even mirrors. Many have 6 inch heels. Or even higher. Gorgeous.

But of course most of these shoes you could never even wear — and not because there’s only one of them. These shoes don’t even make sense as shoes. What ultimately matters is that you can’t do what you need to do with shoes which is walk in them.

Many times I see benchmark comparisons that don’t really focus on the right things as well. Here’s why in comparisons of systems, cores ultimately matter:

  • Cores are the processing units for computation.
  • Cores are used to charge for software licensing.
  • Cores represent a more apples-to-apples method of comparing systems of varying technologies.
  • The right Cores enable efficient virtualization and consolidation which ultimately leads to better total cost of ownership.

Interesting that when these facts are so clear that Oracle’s newest ad on the front page of the Wall Street Journal totally ignores processor cores and many other important components in the comparisons. As you look at the SPECjEnterprise2010 comparisons, here is what you need to know:

  • The IBM benchmark result is from 2012, the Oracle result is brand new. As we know, this is a lifetime of difference for benchmarking.
  • Oracle needed 4x the number of processing cores and 3x the amount of memory than IBM for this benchmark. See all the details here and here.
  • The IBM POWER7+ Power 780 actually has over 1.5x more performance per core than the Oracle SPARC T5 system.(1)
  • Cost is not even a metric of this benchmark. And note that server cost does not include storage and the all expensive software licensing costs, which by the way, are calculated per core.

 

I like shoes and benchmark comparisons which make sense. Give me my New Balance any day. I can walk for miles in them, they look good, and their TCO screams.

Bottom line: Oracle’s latest comparative advertisement targeting IBM Power Systems, like so many before them, strains credulity. Caveat emptor.

************************************************

(1)SPARC T5-8 (8-chip, 128 cores), 27,843.57 SPECjEnterprise2010 EjOPS; IBM Power 780 (8-chips, 32 cores), 10,902.30 SPECjEnterprise2010 EjOPS. Sources: http://www.spec.org. Results current as of 5/23/13.
SPEC and the benchmark name SPECjEnterprise are registered trademarks of the Standard Performance Evaluation Corporation.

The postings on this site solely reflect the personal views of the author and do not necessarily represent the views, positions, strategies or opinions of IBM or IBM management.

technorati tags: , , ,,,,,,,,,,,,,

,,

About these ads

Written by benchmarkingblog

May 23, 2013 at 11:45 am

5 Responses

Subscribe to comments with RSS.

  1. You’ve to backup your blanket statements with some facts. How’s virtualization related to core performance? If it is, then please explain s360 virtualization was possible in the mid-90s when those cores were measly?

    What matters is the TCO; not cores, not power, not multi-threading, not licensing, not virtualization … We care about how I can deliver the most efficient business services and none of these in isolation can deliver that. Explain how core performance will help me in map-reduce world where the strategy is all around splitting workload and sending it to lot of processing units?

    BTW, how’s that F1 race car working out for hauling groceries? Similarly, I wouldn’t take a school bus to a car race either.

    BTW, I’d love lot of infinitely fast processors/cores. I’m not opposed to fast cores, just your case for a few fast cores.

    Anantha

    May 25, 2013 at 1:48 pm

  2. Anantha – My take on the authors statements are to not allow those vendors with inefficient platforms define performance or somehow assert they necessarily outperform, have a lower TCA or a TCO than IBM’s Power family.

    Power servers with both the processor architecture and the Power Hypervisor (PHYP) for virtualization are the secret sauce that provide performance and throughput across the entire portfolio. Power is about flexibility and efficiently.

    Power7 servers (current generation is currently called Power7+) offer servers from 4 – 256 cores ranging from 3.0 – 4.42 GHz. Each core has 4 threads called SMT. PHYP allows the user to divide up the cores as granular as on a single core boundary and allocated up to 20 VM’s per core. Conversely, you could have 1 VM in a single core with 20 virtual processors and with STM4 it would have 80 logical processors. . This means you can have VM’s (aka LPAR) with dedicated cores and/or create multiple Shared Processor Pools (SPP). You can have as many SPP as you want up to the number of active cores. A 4 core server could have 1 SPP with 4 cores or 4 SPP, each with 1 core – you get the idea.

    The next thing you can do on ALL of these Power7 servers is move cores between workloads dynamically – not just cores but also memory and I/O. Yes, add and remove!

    On any given Power server I can allocate physical cores to a workload providing 100% of it’s resources running at up to 4.42 GHz (even higher with its energy management technology). I can concurrently create SPP alongside these dedicated VM’s. These SPP can have a few VM’s or as many as 20 times the number of cores in each SPP.

    The VM’s in those SPP have several configuration options for the cores – they can be fully shared so all un-utilized resources are available to each other or you can limit and protect the amount of resources in the SPP so it receives its processor (processor used generically here as it could a virtual or logical processor based on that VM’s settings) entitlement while still letting the other VM’s in that SPP use it’s un-utilized resources when available – just absolute efficiency isn’t it?!

    This is why Power technology is awesome in that it is all about flexibility. You can put up great benchmark (i.e. raw performance) numbers so you can make marketing claims like the other vendors. But, for customers what matters is not only their perceived TCA but ultimately the TCO.

    Power7 servers come in 2U rack servers and up to 16U for the 19″ rack solutions and the large double wide refrigerator size 795 with 256 cores. These servers are available as inexpensively as roughly $6,000 for the entry level. IBM is currently stating a 16 core 32 GB RAM Power7+ 730 is roughly $12,000. About the same price as a comparable x86 server without virtualization like VMware. This is why claims by other vendors and fanboi’s that Power is so expensive simply do not understand the technology nor IBM’s product offerings. They continue to live in their Big Iron (Itanium & Sparc) worlds or their x86 bubbles of how Power was priced prior to Power7 when the Big Iron wars were still active.

    With Power7 you can now buy entry level servers at x86 pricing while only allocating the cores needed for the workload. Many software products are licensed on the cores required. Now, you can restrict the cost of software licensing as well. Take Oracle for example. For x86 you license the entire server times .5. A 16 core server is 8 Oracle licenses for Enterprise Edition at $47,500 per core even if it is just 25% utilized. Let’s say I can run that same workload with 3 Power7 cores. For Power servers you only need to license the cores needed and NOT the entire server like x86. With a licensing factor of 1.0 that would be 3 cores times $47,500. Take the low TCA I stated above with the lower software licensing costs and over a 3 – 5 year period the savings with Power over x86 could be $500,000. Remember with Oracle you buy the license and pay 22% for the first years maintenance then 22% thereafter. For Power this would be (3 x $47,500 * 1.22) + the 22% of the 1st year license cost. The x86 5 year cost would be (8 x $47,500 * 1.22) + the 22% of the 1st year license cost. (see math below)

    Power: $142,500 + 31,350 = $173,850 TCA; ($31,350 * 4 = $125,400) + 173,850 = $299,250 TCO
    x86: $380,000 + 83,600 = $463,600 TCA; ($83,600 * 4 = $334,400) + $463,600 = $798,000 TCO

    Your premise that TCO is not based on cores, power, threading, licensing, virtualization, etc makes no sense. What is the TCO then? TCO is the TCA of the solution (hardware + software), ongoing maintenance costs, power & cooling costs, data center rack space and the number of FTE (full time equivalents). The more you include the better the Power story gets.

    I don’t know about the F1 car analogy you used but I like to say with Power you can buy a Toyota Yaris, Corolla, Camry, Avalon or jump up to the Lexus all while receiving reliability, performance and efficiency – Power technology is simply about flexibility!

    Brett Murphy

    June 22, 2013 at 3:50 pm

  3. Its great to see IBM refuting comparisons claims, but if you look at the comparison FACTS, its not just about “does the shoe fit” but what “type” of shoes are being compared. In this case, its comparing Oracle’s racing shoes against IBM’s clogs.

    After all, Oracles SPARC T5-8 system with 8 x SPARC T5 CPUs is 3.35x faster than IBM’s *fastest* Power780+ configured system which also has 8 x Power7+ CPU’s. And both systems require 8 Rack Units of server space.

    And while IBM may have chosen its highest frequency Power7+ @ 4.42GHz to show its highest performance/core, it sacrificed 3x lower system throughput using minimal 4-core CPUs, 4x fewer cores/CPU than Oracles SPARC T5.

    So while IBM may be able to claim just 20% faster/core performance (not 50% like IBM claims – SPARC T5-8 w/128-cores=36,571 EJOPs = 285.7 EJOPs/core vs IBM Power 780+ w/32-cores = 10,902 EJOPs = 340.7 EJOPs/core), the system costs alone are 2.3x higher (SPARC T5-8 = $268,054 vs IBM Power 780+ @ ~$610,688).

    And while IBM claims lower licensing costs, that’s actually quite ridiculous as Oracle SW is licensed at a .5x multiplier for SPARC while it’s a 1.0x multiplier for Power7+ so therefore, SPARC T5-8 w/128-cores= 36,571 EJOPs = 64 x licenses = 571 EJOPs/license vs IBM Power 780+ w/32-cores = 10,902 EJOPs = 32 x licenses = 340.7 EJOPs/license, so therefore even on licensing, the SPARC T5 offers 68% higher performance/license.

    Time to look at the devil in the details. SPARC T5 wins. Just like Oracle says.

    Phil Dunn (@Phil_Oracle)

    November 4, 2013 at 9:49 am

    • My last post was 4 months ago. The Oracle brain trust must have been busy solving world hunger or better yet you have been helping our President with his ACA messaging. I can hear Larry now “Phil, we need to help BO with his ACA messaging – what do you have? How about you tell them they can keep their doctor if they want to!”. Larry snorts and says “Great job Phil, now let’s go after that Murphy guy. What do you have? Well, how about we misdirect, mislead then mock him?” Larry couldn’t be prouder!

      Seriously though, Sun and now Oracle has played the chip game for years. You say processor when others mean cores. You conveniently want to compare the T5 and the 780 as 8 cpu systems. Customers don’t pay Oracle Enterprise Edition licensing on chips so nice try. They pay on cores or named users. Let’s focus on cores since that is the topic of discussion. The fact you would even try to claim the T5-8 is 3.35X faster than a 780 based on 8 chips is misleading at best. I actually think you are purposefully being deceptive. The models discussed are the T5-8 with 128 cores vs the 32 core 780 (capable of 64 cores @ 4.42 GHz or 128 cores at 3.7 GHz). You are not faster by any “meaningful” measurement.

      You then try to say IBM sacrificed throughput by using a 4 core cpu in this model of server…so sayeth Oracle? I don’t have time, space or energy to detail the Power architecture here but this has no bearing on the discussion. Quick observation though. Oracle seems intent on building 128, 192, 384 and 512 core servers….wow, is the market for these servers that big? Must be….how about you build a kick-ass 8, 16, 24 and 32 core server? A awesome 2 socket server – as many cores as you can stuff in there or 16 cores in as many sockets as it takes. But, whatever it is, awesome, class leading per core performance. Then your .5 licensing factor would mean something.

      In the 2nd para you say the T5 is 3.35X faster than Power but then seem to acknowledge the Power is at least 20% faster per core. Wow, that must have hurt. Can’t tell if you are coming or going though with your facts – you are faster then you are slower.

      Lastly and again – this is a open letter to customers “Don’t let Oracle continue to mislead you on licensing their software. Power servers are the only platform, whether RISC or x86 that control software licensing costs. You do not license the entire server but only the cores required. The servers are far more efficient than other servers and all are 100% virtualized from birth using hypervisor technology built on mainframe technology. Oracle will say their licensing factor of .5 and Power of 1.0 means Power is twice as expensive. However, here are examples of workload consolidations I have done for customers from SPARC, Itanium, x86 and AMD onto Power technology. 235 cores to 64 Power7 cores. 300 cores to 28 P7 cores. 14 cores to P7 2 cores. 16 cores to P7 4 cores. 532 cores to P7 40(ish) cores – makes you wonder how Larry afforded the boat, the island, the airline, the Malibu strip of 9 homes, the boat, mansion after mansion, etc with all of my hard work :) Yeah, I’m sure you can’t fathom how Power servers do this….because you don’t understand the value proposition of Power technology. SPARC is simply not in the same league.

      My earlier post states the software licensing savings. When Power reduces the Oracle licensing like it does above then customers associated maintenance which is 22% of that number is also significantly reduced. Some customers choose to renegotiate their contracts. Others keep their software investment and never have to buy new license and are able to continue to redeploy the savings from what they have put on the shelf. Customers are free to contact me at my company link to hear the truth – get set free from oppressive Oracle licensing.

      Brett Murphy

      November 5, 2013 at 12:03 am

  4. Well Brett, seems you are betting on the wrong horse.

    Last time I checked, regardless of who you talk to, customers buy systems, they don’t buy cores, and yes I admit, they don’t buy processors either.

    Why does IBM talk cores? Because *all of* IBM’s pricing is based on the core, so therefore it must always show best performance/core, but at the end of the day, its system performance, including throughput, response times and most importantly, price/performance, and especially TCO that matters.

    Heres a good example:
    OLTP Database Performance – show me what you got!

    Oracle SPARC T5-8 with full 128-cores benchmarked: 8,552,523 tpmC @ 0.55 USD per tpmC
    http://www.tpc.org/tpcc/results/tpcc_result_detail.asp?id=113032601

    Avg Response times from 0.014s to 0.254s -Details located here:
    http://www.tpc.org/results/individual_results/oracle/oracle_sparc_t5-8_tpc-c_es_032613.pdf

    IBM’s latest/ fastest non-clustered Power7/Power7+ result:

    IBM Power 780 Server Model 9179-MHB with just 8 x cores? HUH? This system scales to 128-cores today yet IBM publishes just 8 x cores?
    http://www.tpc.org/tpcc/results/tpcc_result_detail.asp?id=110041301

    OK, so IBM says, wow, look at that performance/core-it’s the best! It beats SPARC!

    So then you look at the throughput, Price/Perf and response times. Located here:
    http://c970058.r58.cf2.rackcdn.com/individual_results/IBM/IBM_780_TPCC_20100719_es.pdf

    IBM Power 780 Server Model 9179-MHB : Only 1,200,011 tpmC and 0.69 USD per tpmC
    Avg Response times from 0.234s to 0.421s

    So that’s 7x lower throughput than the Oracle SPARC T5 published result, 25% worse price/performance and 66% to 16.7x worse response times. Hmm, yet IBM states it has 2x higher performance/core?

    So bottom line, there is no public benchmark that has a performance/core metric? Why? Because performance/core changes depending on the system configuration and typically, the more cores you have in a system, the worse the performance/core calculation becomes. That’s why you’ll never see an IBM benchmark published with maximum possible configured # of cores, unless it’s a benchmark that doesn’t stress the full system running a real world workload. Even using IBM’s own rPerf metric will show worse performance/core as you scale to higher # of cores in system.

    And if you look at *any* public benchmark where there is a price/performance metric, find me just one, where Power7 (it’ll be tough to find any benchmarks with Power7+) shows better than SPARC T5 and that shows better throughput, better response times or even price/performance. TPC-C, TPC-H, SPECjEnterprise2010, SPECjbb2013, STREAM, JD Edwards, Siebel, SAP, etc. Yeah, I know you’re response. You’ll say Oracle tricked the benchmarks. Actually, Oracle did. They ran Oracle on Oracle and showing the benefits which you just can’t get on other platforms.

    Oh and by the way, IBM hasn’t published SPECjbb2013 yet? Why not? Java not an important workload for Power7/Power7+? Websphere anyone? This is what customers care about.

    And as for virtualization, Oracles VM technology is built in to SPARC and theres also Solaris zones to allow customers to reduce license requirements and maximize license utilization. These have identical functionality and capabilities to what IBM offers.

    So the bottom line, is if you want to run Oracle SW, theres no better, lower cost, higher performance platform to run it on than Oracle SPARC. And until IBM publishes *any* benchmarks to prove otherwise, what you’re saying is just fluff. And as for Larry, the reason he is where he is its called “success” and #1 leadership for a very long time. Something we all envy and strive for.

    Phil Dunn (@Phil_Oracle)

    November 5, 2013 at 6:04 am


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: