Performance

Sun Fire 15K[tm] 6,400 Mainframe Equivalent MIPS Ratings

The Sun Fire 15K server has more computational power than the most powerful mainframes. Mainframe performance is often compared using the MIPS performance metric. Sun estimates the Sun Fire 15K[tm] server performance to be equivalent to 6,400 Mainframe MIPS. This rating is based on head-to-head comparisons of benchmarks run on the 72-way Sun Fire 15K and the 64-way Starfire[tm] and on various IBM mainframes. To obtain an accurate estimation, Sun compared a diverse set of workloads including results on SAP, data warehousing, CICS, and customer applications, and IBM's own iSeries MIPS estimates. In each case, MIPS estimates are very close helps verify the precision of this methodology. Sun has been publicly estimating mainframe equivalent MIPS ratings on the Sun Starfire since 1997.

Table A: Sun Fire 15K & Starfire Servers vs. IBM Mainframes (sorted by MIPS)

Table A Sources:

• Starfire Server Figures:  Calculated from benchmark figures detailed in the sections below 654=2059990/3150 (64-way 64GB); 562=3594000/6400 (72-way 144GB)
• IBM zSeries Figures:  http://www.tech-news.com/publib/pl2064.htm $1801/MIPS = $4851000/2694 MIPS
• IBM G6 Figures:  http://www.tech-news.com/publib/pl9672g6.htm $2945/MIPS = $4842000/1644 MIPS
• IBM G5 Figures:  http://www.tech-news.com/publib/pl9672g5.htm $3742/MIPS = $4000000/1069 MIPS
• IBM introduction dates provided by Bernstein & Co. LLC

The summary of these comparisons are shown in the SAP, DSS, UniKix, and OLTP benchmarks below.

Table B: Sun Mainframe Equivalent MIPS Summary:

(Note: see below for all calculations that were used to derive these figures,
* denotes estimated by scaling result by 20% since 400Mhz is 20% faster on system Mhz and CPU MHz).
SAP SD 336Mhz 2552 MIPS *1.2 = 3062 MIPS* for 400Mhz
OTLP IO 336Mhz 2454 MIPS *1.2 = 2945 MIPS* for 400Mhz

Methodology:

Sun has a long history of estimating Mainframe equivalent MIPS based on comparing application performance run on mainframes and on Sun servers. This MIPS performance rating is derived by both by taking the application performance on a mainframe of known MIPS rating and comparing the same workload on Sun Servers. Then ratios are constructed to estimate the Mainframe equivalent MIPS for Sun servers. We have found that each workload comparison yields approximately the same MIPS rating. We have used this methodology for multiple generations of Starfire servers.

In this analysis, Sun has tried to use every publicly available IBM performance benchmark with sufficient description to allow new comparisons. In addition, this is cross-checked with the relative performance difference between Sun server generations. Because Sun has measured the performance between Starfire server and the Sun Fire 15K server we know how to estimate the performance improvement in MIPS. It has become increasingly difficult to compare to IBM mainframes due to IBM's lack of publicly verifiable benchmark results on the zSeries.

The Sun Fire 15K Mainframe equivalent MIPS was calculated in the same manner. In addition, we now have the MIPS rating for other IBM systems that we have directly benchmarked against. This new analysis is below.

MIPS Performance Comparisons

iSeries (AS/400) MIPS and the Sun Fire 15K Server MIPS

IBM's AS400 chief scientist, Frank G. Soltis estimated MIPS rating of the IBM i840 600Mhz at 2,800 MIPS. Previously this IBM product line was known by the name AS/400. Soltis explores the IBM i840's performance characteristics in the article:

Soltis, Frank G., "Performance Comparisons: Why Must They Be So Hard?", iSeries News, Penton Publication, January 2002, pp. 21-27. http://www.iseriesnetwork.com/Article.cfm?ID=12036

Since Sun and IBM have both run the equivalent SAP-SD OLTP database benchmark we can use Soltis's work to estimate the MIPS rating for the Sun Fire 15K server.

As the article states, "IBM doesn't even disclose the actual calculations that are used to determine each of these metrics (MIPS, rPerf, CPW), which makes it practically impossible to compare across the product lines." Nevertheless using as direct a comparison as possible, IBM's Soltis decided to estimate performance using a variety of techniques.

"We (IBM) decided to use the same calculations that are used to determine mainframe MIPS to see how an AS/400 compared. To generate the MIPS number, we look at the mix of instructions that are used in a "typical" workload and measure what percentages of the total number of instructions are loads, stores, branches, and so on."

Using this methodology in May 2000 on 24-way AS400 (I-Star processors), IBM estimated the performance at approximately 2,200 MIPS. It is interesting to note that IBM stopped publicly talking about this estimate as they realized it eclipsed their largest mainframe at the time. At the end of the article, IBM's Soltis estimates that a current and faster 24-way i840 (600Mhz AS/400) has a performance rating of 2,800 MIPS.

SAP comparison

An OLTP database benchmark that has been run on the Sun Fire 15K server and the IBM i840 (600Mhz) is the SAP SD 2-tier benchmark version 4.6C.

In this test the Sun Fire 15K server did more than three times the number of line items/hour (the measure of speed) than the IBM i840 server. In other words the 76-way system was 3.05 times faster (3.05=414,000/135,670 LI/hour). Scaling this down to a 72-way Sun fire 15K server we expect it to be 2.89 times faster (2.89 = (414,000/135,670)*(72/76)).

iSeries CPW Performance vs. Sun Fire 15K System

IBM rates the performance of the iSeries using CPW (Commercial Processing Workload). The 500 MHz i840 is rated at 16,500 CPW, and the 600 MHz i840 is rated at 20,200 CPW. Soltis goes on to say, "In fact, the CPW ratings we use for all iSeries servers are measured using a benchmark that closely resembles the TPC-C benchmark."

Above we showed that the 72-way Sun Fire 15K server running on an SAP OLTP workload was approximately 2.89 times faster than the 24-way i840, therefore we estimate the Sun Fire 15K has a CPW of 58,378 = 20,200*2.89.

Conclusion

Sun found that the Sun Fire 15K server was 2.89 times faster than the IBM i840 on the same audited SAP SD 2-tier OLTP benchmark. This comparison would rate the Sun Fire 15K server at 8,092 MIPS, which is more than the conservative 6,400 Estimated Mainframe MIPS that Sun rated the Sun Fire 15K server at product introduction using other benchmark comparisons. To be conservative, Sun will continue to use 6,400 Mainframe Equivalent MIPS for the Sun Fire 15K server.

SAP SD 3-Tier Benchmark

The SAP R/3 SD Benchmark is a publicly scrutinized high-volume OLTP benchmark with very precise benchmark rules. It forms the closest comparison of mainframe to Starfire server performance.

Table C: SAP Benchmark Comparisons

Table C Legend/Sources:

• SD = SAP "SD" Benchmark
  
• SDP= SAP "SD Parallel" Benchmark (Results are not directly comparable to the SD Benchmark)
  
• (1)IBM S/390 G5 10-way estimates; we estimate 6375, using 6.0
  users/MIPS. IBM says that the 10-way G5 would support 6400 SD Benchmark SAP users (see http://www.techweb.com/wire/story/TWB19980508S0028).

All other IBM and Sun SAP R/3 benchmark results in Table B can be found on SAP's website, http://www.sap.com/benchmark/ under R/3 Benchmark Results.

IBM Sysplex can be used to "cluster" more mainframes together, but Sysplex users are told to expect around 10% overhead (See "IBM's S/390 Parallel Sysplex Overhead: A Reality Check" http://www.S390.ibm.com/marketing/gf225009.html). Note that two comparable benchmarks shown here -- SAP and DSS -- have 24% to 33% overhead.

Starfire estimates based on the Throughput Metric (Dialog Steps/hour)

Table D: Throughput Metric Comparisons

Table D Legend/Sources:

• Starfire server estimate 336 MHz ((4,725,000)/81.8%))/(2263.6 dialog-hour/MIPS) = 2552 Mips
• Starfire server estimate 250 MHz (((1,820,000)/(33 CPUs*86%))*64)/(2263 dialog-hour/MIPS) = 1813 MIPS
• IBM and Sun benchmark results detailed in Table C can be found on the SAP website, http://www.sap.com/benchmark/ under R/3 Benchmark Results.
  
  For many applications we have found that a 400 MHz Starfire system is 20% faster than the Starfire running at 336MHz.

DSS Performance

IBM's website provides information on mainframe DSS performance that allows us to draw a couple of comparisons and estimate mainframe MIPS.

Table E: DSS Performance Comparisons

Table E Legend/Sources:

• Sun Fire 15K server estimate using worst case 1.94 MIPS/MB/sec from above (3429*1.94)=6652 MIPS
• Starfire 400 server estimate using worst case 1.94 MIPS/MB/sec from above (2011*1.94)=3901 MIPS
• Starfire 336 server estimate using worst case 1.94 MIPS/MB/sec from above (1450*1.94)=2813 MIPS
• Starfire 250 server estimate using worst case 1.94 MIPS/MB/sec from above (1092*1.94)=2118 MIPS
• IBM Sources...
  http://www.S390.ibm.com/marketing/s390query.html "Sizzling S/390 Query Performance," GF22-5023-00
  http://www.S390.ibm.com/marketing/sparkling.html "Investment Grade S/390 Query Solutions," GF22-5024-00

The table above shows an IO intensive full table scan query. This is directly comparable to a full table scan shown on the Starfire server. In fact, both the mainframe and the Sun runs were done on 750 GB tables with 6 Billion rows.

The other IBM S390 test was a full scan of a 180 million row, 23 gigabyte table using the new DB2 V4 "parallel" query, running on a 10-engine IBM S390 9672-RX3 with ten 3990 storage controllers, 40 data partitions. The full scan on DB2 V4 takes 4.6 minutes. This is only 85 MB/sec for this 165MIPS Rx3! Which is about 1.94 Mips/MB/sec. This was done on a TPC-D schema on a SF30 data size.

SPEC WEB96 Performance on Mainframes

IBM is positioning Mainframes for webserver consolidation. At one time IBM posted results on SPEC benchmarks using the mainframe, however no new SPEC or TPC results have been published on mainframe systems.

One CPU of the G5 mainframe rated at 152 MIPS was only 9% faster than a 400Mhz UltraSPARC system. Using the ratio of performance above a 400Mhz CPU would be 139 MIPS. In this generation of Sun CPUs we rate a Starfire at 3000 MIPS for 64 CPUs (46.8 = 3000 MIPS / 64 CPUs).

This calculation is not used above calculation of MIPS, but it does support that Sun processors can be very efficient when compared to mainframes.

Information Source:
http://www.specbench.org/osg/web96/results/res98q4/

SPEC and SPECWEB96 are registered trademarks of the Standard Performance Evaluation Corporation. The SPECWEB96 benchmark was retired SPECweb96 results are no longer being reviewed or published by SPEC. For more information on SPEC results, visit http://www.spec.org/osg/web96/.

UniKix Performance

CICS performance is also a concern for mainframe sites, as pointed out in the Feb 1998 UNIX Reseller article, "Total Cost Of Ownership" (formerly located at http://www.unixreview.com/articles/1998/9802/9802res/9802res.htm) in which a comparison is made between the performance of Mainframes and UNIX systems by comparing UniKix to CICS on over 100 customer workloads.

The article concluded that 1 SPECint95 was between 4.1 and 10.2 mainframe MIPS (6.4 average). Given a conservative 16.8 SPECint95 per Starfire 400 Mhz CPU and a very conservative 4.1 MIPS/SPECint95 we can estimate of 16.8*64*4.1 = 4408 MIPS for the Starfire server. Likewise, given a conservative 14.0 SPECint95 Sun arrives at 3673 MIPS for the 336Mhz Starfire server and given a conservative 10.2 SPECint95 per Starfire 250 Mhz CPU Sun arrives at 2676 MIPS for 250Mhz Starfire.

General OLTP

For general OLTP, a variety of benchmarks Sun estimates that 33 physical IO/sec are supported per mainframe MIPS in an OLTP workload. Given a delivered (but not maximum) of 81,000 IO/sec we arrived at a figure of 2454 MIPS for a 64 CPU 336 MHz Starfire server. Alternatively we delivered 66,000 IO/sec on a 250MHz 64 CPU Starfire for a MIPS rating of 2078 MIPS. Many customers are amazed when they compare total physical IOs that a mainframe delivers to what is delivered on the Starfire server.

For many applications Sun has found that a 400 MHz Starfire system is 20% faster than the Starfire running at 336MHz.

General Notes:

1. In http://www.S390.ibm.com/marketing/gf225009.html IBM estimates 10-20% overhead due to Sysplex. "Bottom line, amortized over the entire sysplex, a typical reported average is around 10%." In the SAP benchmark detailed on this page they showed approximately a 33% degrade.
   
   
2. IBM has few industry-standard benchmarks that are comparable and available for public scrutiny.
   
   
3. IBM routinely claims that 32 systems can be put together in a Sysplex with 320 processors, to our knowledge no benchmark has been run on this configuration.
   
   
4. There are other methods to compare mainframe performance, most of these can be directly related to MIPS. Most users traditionally think of mainframe performance capacity in terms of MIPS and that is why they are used here.

This document has been reviewed by SMI Marketing Legal and is approved for DISTRIBUTION WITHIN THE UNITED STATES (EXCLUDING ITS TERRITORIES) AND CANADA ONLY. For distribution outside these specific regions, an in-country legal opinion must be obtained.