MICROPROCESSOR

REPORT www.MPRonline.com

T H E I N S I D E R ’ S G U I D E T O M I C R O P R O C E S S O R H A R D WA R E

SPEC CPU2006 BENCHMARK SUITE

The Newest Iteration of a Widely Used Measure of Processor Performance
By Har lan McGhan {10/10/06-02}

On August 24, 2006, the nonprofit Standard Performance Evaluation Corporation (SPEC)
announced the release of its new SPEC CPU2006 benchmark suite. This is the fifth iteration
in a line of SPEC CPU benchmark suites that began in 1989 and continued with new versions

in 1992 (CPU92), 1995 (CPU95), and 2000 (CPU2000). The
roughly five-year lifetime of a SPEC CPU benchmark suite now
seems to be established policy, which is to say that we can expect
CPU2006 to be the standard from now through at least 2011.
Benchmarks tend to arouse strong passions among
those with a professional (or monetary) interest in com-
puter performance. Many veterans of the performance wars
believe that benchmarks tend to conceal and distort far
more than they ever reveal or reflect. (See MPR 6/26/00-01,
“Benchmarks Are Bunk.”) Fortunately, it is not necessary to
first agree on the merits of benchmarking before agreeing
that the SPEC CPU benchmark suite is important.
SPEC CPU is by far the most widely used measure of
processor performance in history. The SPEC CPU bench-
mark programs themselves are among the most extensively
studied programs in the computer industry, the primary
target of many processor architects, compiler writers, com-
petitive analysts, and academic researchers—to name only
some vitally interested parties. A new iteration of the SPEC
CPU benchmark suite, therefore, carries weight. It is sure to
affect the entire industry—for good or ill—in many ways.
Given the length of time that has elapsed since its predeces-
sor suite was released (see MPR 4/17/00-02, “SPEC
CPU2000 Released”) and the significant changes that have
occurred in the processor landscape since 2000, this latest
version of the SPEC CPU benchmark suite merits careful
consideration. (See sidebar on “Price and Availability of
SPEC CPU2006” for product details.)
Nature of the SPEC CPU Benchmark Suite
SPEC was an industrywide response to an industrywide
problem that reached a crisis point during the “processor
wars” of the late 1980s: namely, the lack of any generally
acceptable means of comparing processor performance
across widely different instruction-set architectures (ISAs—
see sidebar, “The Problem With MIPS”). To solve this prob-
lem, the leading companies of the day (including DEC, HP,
IBM, Intel, MIPS Technologies, Motorola, SGI, Sun) banded
together to create the SPEC consortium. SPEC’s goal was to
develop a robust cross-platform suite of representative test
programs, to create methods for converting this test suite
into meaningful scores, and to establish fair rules for run-
ning benchmark programs and reporting benchmark scores.
The focus of SPEC CPU is on “component,” specifically
processor (CPU), performance rather than “whole system”
performance. (The capital C prefixed to the names of SPEC
CPU tests, e.g., Cint2006 and Cfp2006, stands for compo-
nent.) One charge sometimes lodged against SPEC CPU
scores, therefore, can be readily laid to rest: namely, the objec-
tion that SPEC CPU scores measure only some aspects of a
system’s performance. (And—for anyone who has ever been
stuck on the wrong side of a slow disk drive or a slow network
connection—by no means the most important aspect.) This
objection is absolutely true and also completely irrelevant.
SPEC CPU is intended to measure only one aspect of com-
puter performance: specifically, basic processor “horsepower.”
Admittedly, many other aspects of a system also contribute to

© I N - S TAT OCTOBER 10, 2006 MICROPROCESSOR REPORT

 2 SPEC CPU2006 Benchmark Suite

the overall performance that system delivers. Indeed, over
the years, SPEC has formed a variety of committees to
develop generally acceptable benchmarks for other impor-
tant aspects of computer performance, including graphics,
as well as for various specialized types of “whole” systems,
including Java client servers, mail servers, web servers, and
network file servers.
None of these other tests, however, tells us anything
(much) about processor performance, just as SPEC CPU
tells us nothing (much) about graphics or disk or network
I/O performance. If processor performance is your interest,
you need a spotlight to pick it out of the general landscape.
The specific intention of SPEC CPU is to provide that spot-
light (leaving all else in shadow).
From this perspective, it is regrettable that SPEC CPU
is not even more specific than it is or can be. To support
SPEC CPU’s cross-platform requirement, programs in the
SPEC benchmark suite have to be supplied in source form.
Consequently, the minimum component testable by these
programs on any given machine includes the compiler that
converts these programs into (more or less) efficient
machine code. Also, since few realistic programs fit entirely
into a processor’s on-chip cache memory (which also varies
greatly in size and configuration from processor to proces-
sor), the SPEC committee broadened its definition of the
basic processor component to include both off-chip cache
and main memory (but to exclude disk drives, network stor-
age, and all other I/O operations).
This notion of the “processor” component (encompass-
ing everything from the compiler to main memory) sets the
limits for a suitable SPEC CPU benchmark program. First, it
800
700 41.67MHz POWER 4164
64K/8K I/D Caches
SPEC Performance Ratio
must be written in a “neutral” way (favoring no special fea-
tures of any particular ISA or microarchitecture), using a
“standard” programming language (i.e., one for which good
compilers readily exist on all platforms). To date, this means
that SPEC programs have been limited to just C, C++, and
(for floating-point-intensive programs) Fortran (both For-
tran77 and Fortran90 have been used). Second, to test all and
only the relevant parts of the memory hierarchy, the bench-
mark program must compile to a working set too large to fit
into any available (on- or off-chip) cache but small enough to
fit into a typical main memory.
Why a New SPEC CPU Benchmark Suite?
The fact that the typical size of both cache and main mem-
ory for processors continually increases is one important
reason that the SPEC CPU benchmark suite must be revised
periodically. For example, when SPEC CPU2000 was intro-
duced, the size of a typical on-chip cache was 256KB (or
less), the largest off-chip cache was 8MB, and a reasonable
minimum size for main memory was 256MB. In 2006, typi-
cal on-chip caches range up to 4MB, the largest off-chip
cache is 36MB, and a reasonable minimum main memory
size is 1GB. Programs that once fairly tested the largest mem-
ory hierarchies of 2000 are now too small to realistically
exercise normal memory hierarchies—hence, the debut of
SPEC CPU 2006, replacing SPEC CPU 2000.
Just as the size of typical memories increases over time,
so does the power of typical processors. In 2000, the fastest
processor was Intel’s just-introduced Pentium 4 at 1.5GHz.
In 2006, Intel’s latest Pentium 4 Extreme Edition operates at
3.73GHz (with the company’s even more powerful next-
generation Core 2 Extreme processor just coming
into use). SPEC CPU2000 programs that once took
hundreds of seconds to execute on a 1.5GHz Pen-
tium 4 now execute in a few tens of seconds on
these newer machines. With such short run times,

600 64MB Memory

AIX 3.1.5 Operating System
500
400
300
200
100
0 Improvements in compiler technology also
gcc espresso spice doduc nasa7 li eqntott matrix300 fpppp tomcatv
Benchmark
Figure 1. 1991 SPEC CPU89 results for the IBM PowerStation 550 (based on the
41.67MHz POWER 4164 processor) with and without enhancing compilers.
Although use of a geometrical mean average, used for computing overall SPECmarks,
smooths out disturbances caused by atypical results on individual component pro-
grams, even a geometric mean couldn’t average away the effect of the extreme
increase in matrix300 performance caused by newfound ways to optimized this com-
ponent benchmark. The SPEC committee dropped matrix300 from the 1992 test
suite. The nasa7 component, however, survived until the 1995 test suite (despite the
ability of optimizing compilers to significantly enhance its performance, too).
results are increasingly perturbed by minor fluctu-
ations in the overall system state or measurement
circumstances. In short, many older programs sim-
ply don’t run long enough to draw accurate con-
clusions from them. Correcting this problem
requires not only bigger programs but more com-
plex programs, as well as programs with much
larger working sets that take far longer to complete.
can obsolete a particular benchmark. The first and
still one of the most famous examples of this type
of problem occurred with the matrix300 compo-
nent program in the original 1989 benchmark
suite. In 1990, it made little difference whether a
SPECmark was computed with or without this
component program. A year later, it made all the
difference in the world. For example, in 1990, the
30MHz IBM POWER 3064 achieved a SPECmark
of 34.7; recomputing this value omitting

© I N - S TAT OCTOBER 10, 2006 MICROPROCESSOR REPORT

 SPEC CPU2006 Benchmark Suite 3

matrix300 resulted in a slightly higher overall score of 35.7.
Just a year later, in 1991, the 41.67MHz POWER 4164
achieved a SPECmark of 72.2; recomputing this value omit-
ting matrix300 resulted in a dramatically lower score of 55.9.
So how much did the performance of IBM’s POWER
processor line improve in 1991 from 1990? The matrix300
program looms large in the answer. Counting it, the improve-
ment was over 108%. Excluding it, the improvement was less
than 57%. The explanation of this phenomenon is that, when
introduced in 1989, matrix300 was (by intention) a cache
“thrasher” program, requiring constant trips to main mem-
ory. Hence, it ran poorly, meaning that omitting this compo-
nent test was likely to improve a processor’s overall SPECmark
score. In 1991, compiler writers (or, more precisely, compiler
preprocessor writers) discovered how to “bust” this program,
devising a scheme for organizing its data references so that the
program ran almost entirely in cache. The result was like
throwing a switch. Scores on this one component program
suddenly rocketed through the roof, taking the machine’s
entire SPECmark score up with them. For an illustration of
the dramatic difference caused by introducing this new opti-
mization, see Figure 1.
Although an extreme example of the effect of compiler
(or precompiler) optimization, matrix300 is by no means an
isolated example. An unintended consequence of the rise of
the SPEC CPU benchmark suite to prominence, as the central
focus of the processor performance wars, has been intense
scrutiny of every component program in each successive iter-
ation of this benchmark suite. Both hardware designers and
compiler writers measure their efforts to improve successive
generations of their respective products by the degree to
which they raise SPEC benchmark scores. Occasionally, as
with matrix300, they are inspired to find a new way of attack-
ing a formerly intractable performance issue. While the long-
term effect of this unrelenting attack on knotty performance
issues is generally positive, constantly adding new and refined
software and hardware technologies to the overall perform-
ance arsenal, the short-term effect on the SPEC benchmark
results themselves can be both disruptive and misleading—
until the suite is revised to eliminate the source of the distur-
bance. (For another example of benchmark obsolescence due
to compiler optimization, see the discussion of the 023.eqntott
component in the SPEC CPU92 suite at www.spec.org/
osg/news/articles/news9512/eqntott.html.)
Still another important reason to periodically refresh
the SPEC CPU benchmark suite is the fact that typical
workloads shift over time, with new types of computations
coming to the foreground. The SPEC CPU2000 benchmark
suite put more emphasis on 3D modeling, image recogni-
tion, and data compression than did the SPEC CPU95
benchmark suite. The new SPEC CPU2006 suite adds video
compression and speech recognition to the mix and greatly
expands the emphasis placed on molecular biology among
the scientific (floating-point-intensive) tasks. See Table 1

CPU2000 CPU2006
Benchmark Description Integer Lng RT Integer Lng RT
GNU C compiler 176.gcc C 1,100 403.gcc C 8,050
Manipulates strings & prime numbers in Perl language 253.perlbmk C 1,800 400.perlbench C 9,766
Minimum cost network flow solver (combinatorial optimization) 181.mcf C 1,800 429.mcf C 9,120
Data compression utility 256.bzip2 C 1,500 401.bzip2 C 9,644
Data compression utility 164.gzip C 1,400
Video compression & decompression 464.h264ref C 22,235
Artificial intelligence, plays game of Chess 186.crafty C 1,000 458.sjeng C 12,141
Artificial intelligence, plays game of Go 445.gobmk C 10,489
Artificial intelligence used in games for finding 2D paths across terrains 473.astar C++ 7,017
Natural language processing 197.parser C 1,800
XML processing 483.xalancbmk C++ 6,869
FPGA circuit placement and routing 175.vpr C 1,400
EDA place and route simulator 300.twolf C 3,000
Search gene sequence 456.hmmer C 9,333
Ray tracing 252.eon C++ 1,300
Computational group theory 254.gap C 1,100
Database program 255.vortex C 1,900
Library for simulating a quantum computer 462.libquantum C 20,704
Discrete event simulation 471.omnetpp C++ 6,270
hours 5.3 19,100 hours 36.6 131,638

Table 1. Changes in SPEC CPU benchmark integer programs between CPU2000 and CPU2006. The reference time (RT) for each component is meas-
ured on a reference machine. (A similar, but not identical, near-300MHz UltraSPARC II generation processor is used for both suites.) Although four com-
ponents have been retained from the 2000 suite, the run time for all programs has dramatically increased in CPU 2006. Data compression has been
deemphasized, while video compression and decompression have been added to the test suite. More emphasis is placed on artificial intelligence (includ-
ing a greatly improved chess-playing program). Natural language processing has been eliminated in favor of extended markup language (XML) pro-
cessing. The former emphasis on electronic design-automation programs is gone, as are former programs for ray tracing, computational group theory,
and database operations. In their place is a new emphasis on gene sequencing, quantum computing, and discrete event simulation. More emphasis is
also placed on C++ programming.

© I N - S TAT OCTOBER 10, 2006 MICROPROCESSOR REPORT

4 SPEC CPU2006 Benchmark Suite

and Table 2 for a detailed comparison of the component
programs in the 2000 and 2006 SPEC CPU integer and
floating-point suites, respectively.
Is SPEC CPU2006 Well Balanced?
The most usual objection raised to any new version of the SPEC
CPU benchmark suite is that some key types of programs are
over-represented, while others are under-represented, with cor-
responding skews in the numbers reported. Although the num-
ber of programs included in the SPEC CPU benchmark suite
has gradually crept up over the years (from 10 in the original
CPU89 suite to 18 in CPU95 to 26 in CPU2000 to 29 in the new
CPU2006 suite), there is no way to ever incorporate enough
programs to entirely eliminate this objection. Certainly, the spe-
cific mix of programs in CPU2006 (or any past or probable
future version of the suite) is unlikely to represent the mix of
programs run at any actual computing site. For example, there
are very few Go-playing, gene-sequencing, weather-modeling,
astrophysical quantum chemistry sites.
This particular objection, of course, can be met by sub-
setting the benchmark suite, looking just at those programs
that actually do reflect the types of applications run at a given
site. Thus, a site doing fluid dynamics or quantum chemistry
might look primarily at the specific results reported on
benchmark programs in those particular areas, giving weight
to other components only to the degree that they mirror its
own computational needs.
It is more difficult to meet the objection that some areas
of interest are entirely ignored by the programs included in the
suite. For example, critics complained the CPU2000 suite
dropped JPEG compression (discrete-cosine transformations),
eliminated single-precision floating-point operations, ignored
advanced compression algorithms (e.g., fractals and wavelets),
and omitted speech-recognition technologies. Furthermore,
they argued against the general orientation of the SPEC suite,
leaning as it does toward limited-interest, heavy-duty scientific
applications (run, say, at a site like Brookhaven National Labs
for investigations into quark coloring [chromodynamics] or

CPU2000 CPU2006
Benchmark Description Floating Pnt Lng RTime Floating Pnt Lng RTime
Weather prediction, shallow water model 171.swim F77 3,100
Velocity & distribution of pollutants based on temperature, wind 301.apsi F77 2,600
Weather modeling (30km area over 2 days) 481.wrf C/F 11,215
Physics, particle accelerator model 200.sixtrack F77 1,100
Parabolic/elliptic partial differential equations 173.applu F77 2,100
Multi-grid solver in 3D potential field 172.mgrid F77 1,800
General relativity, solves Einstein evolution equations 436.cactusADM C/F 11,927
Computational electromagnetics (solves Maxwell equations in 3D) 459.GemsFDTD F 10,583
Quantum chromodynamics 168.wupwise F77 1,600
Quantum chromodynamics, gauge field generation with dynamical quarks 433.milc C 9,180
Fluid dynamics, analysis of oscillatory instability 178.galgel F90 2,900
Fluid dynamics, computes 3D transonic transient laminar viscous flow 410.bwaves F 13,592
Computational fluid dynamics for simulation of astrophysical phenomena 434.zeusmp F 9,096
Fluid dynamics, large eddy simulations with linear-eddy model in 3D 437.leslie3d F 9,358
Fluid dynamics, simulates incompressible fluids in 3D 470.lbm C 13,718
Molecular dynamics (simulations based on newtonian equations of motion) 435.gromacs C/F 7,132
Biomolecular dynamics, simulates large system with 92,224 atoms 444.namd C++ 8,018
Computational chemistry 188.ammp C 2,200
Quantum chemistry package (object-oriented design) 465.tonto F 9,822
Quantum chemistry, wide range of self-consistent field calculations 416.gamess F 19,575
Computer vision, face recognition 187.facerec F90 1,900
Speech recognition system 482.sphinx3 C 19,528
3D graphics library 177.mesa C 1,400
Neural network simulation (adaptive resonance theory) 179.art C 2,600
Earthquake modeling (finite element simulation) 183.equake C 1,300
Crash modeling (finite element simulation) 191.fma3d F90 2,100
Number theory (testing for primes) 189.lucas F90 2,000
Structural mechanics (finite elements for linear & nonlinear 3D structures) 454.calculix C/F 8,250
Finite element analysis (program library) 447.dealII C++ 11,486
Linear programming optimization (railroad planning, airlift models) 450.soplex C++ 8,338
Image ray tracing (400x400 anti-aliased image with abstract objects) 453.povray C++ 5,346
hours 8.0 28,700 hours 52 186,164
hours 13.3 47,810 hours 88.3 317,802

Table 2. Changes in SPEC CPU benchmark floating-point programs between CPU2000 and CPU2006. Again, note the dramatic sevenfold increase in run
time of the suite on the (very similar) reference machine. CPU2006 marks the first-ever total revision of a SPEC benchmark suite from one generation to
the next. Although meteorology, physics, fluid dynamics, and chemistry have all been retained as areas of interest, CPU2006 provides new, larger, and
(often) more programs in each field, as well as new programs in the area of molecular dynamics. Face recognition has been eliminated in favor of speech
recognition. Miscellaneous graphics, modeling, and number theory programs are replaced by engineering, route planning, and ray tracing programs. The
variety of languages represented has also been substantially increased, including four programs written in a mixture of C and Fortran.

© I N - S TAT OCTOBER 10, 2006 MICROPROCESSOR REPORT


big bang theory) and away from far more widely relevant
PC-style applications (involving complex floating-point
calculations, advanced game playing, positional 3D sound,
and so on).
To an extent, CPU2006 repairs some of these omissions
in CPU2000. The SPEC committee included both video-
compression/decompression and speech-recognition pro-
grams, throwing in XML (extended markup language) process-
ing for good measure. It also put considerably more emphasis
on artificial intelligence for game playing. But at the same time,
it entirely eliminated EDA (electronic design automation) pro-
grams. The emphasis placed on advanced physics and other
heavy-duty scientific applications is now even greater than it
was in CPU2000, with eleven programs where before there were
only six. Moreover, as measured by the reference machine,
these eleven CPU2006 programs (in physics, fluid dynamics,
molecular dynamics, and quantum chemistry) represent more
than 65% of the floating-point suite’s total run time. In
CPU2000, the corresponding six programs took only 40% of
the floating-point suite’s total run time.
Two Defenses of SPEC CPU2006 Balance
There are two possible defenses to the objection that CPU2006
is an unbalanced mix of programs (an objection, to repeat, that
is doubtlessly unavoidable with any relatively small group of
programs). The first defense is to argue simply that any
machine powerful enough to handle the needs of the National
Laboratory at Brookhaven (Sandia, Oak Ridge, Los Alamos,
Lawrence Livermore, etc.) is probably powerful enough to
handle just about any other conceivable type of need.
The second defense involves shifting the focus away
from the application level to a more basic level of computer
operation. From the standpoint of basic machine code, not
that many different types of tasks are performed by comput-
ers. All programs (regardless of application type) consist of a
mixture of loads and stores, interspersed with sequences of
calculations interrupted by branches. At bottom, the calcula-
tions themselves reduce to combinations of a relatively few
fundamental operations: addition, subtraction, compares,
logical AND, OR, NOT, and so forth—the exact number and
nature of the available operators depending on a particular
machine’s specific ISA. How well any program performs,
then, depends on the number and nature of the loads, stores,
and branches it involves, as well as the specific combinations
of basic arithmetical and logical operations that it performs
(for example, the extent to which these various activities con-
ceal or reveal exploitable instruction-level parallelism).
Over the years, numerous studies have been conducted
of the various SPEC CPU benchmark suites at this more
basic computer science level. All these studies reach a similar
conclusion: namely, that the same overall results can be
obtained by using just a subset of the programs actually
included in the suite. In other words, when analyzed at a fun-
damental level, programs in the suite tend to be less dissim-
ilar from each other than may first appear at the application
© I N - S TAT OCTOBER 10, 2006
SPEC CPU2006 Benchmark Suite 5
level. In truth, rather than too few programs, the several SPEC
CPU suites have each tended to contain too many programs:
that is, they invariably comprehend redundant programs
that add little or nothing to the mixture of operations repre-
sented and whose inclusion or exclusion makes little or no
difference to the overall score achieved.
To the extent that the greatest error of a SPEC CPU
benchmark suite is plain redundancy, it is at least a relatively
harmless flaw. To be sure, running redundant programs adds
time and cost to the SPEC CPU measurements. However, this
is a relatively small price to pay for completeness of coverage.
Far better to err, as the SPEC committee has consistently
done, on the side of over-representing typical workloads than
risk under-representing the range of tasks computers are
called upon to perform.
The SPEC CPU Measures
Although, for all the reasons enumerated above, the basic
thing SPEC CPU must periodically upgrade is the bench-
mark suite itself, both the associated measures and their run
and reporting rules have evolved over time in an effort to bet-
ter capture CPU performance. In 1989, SPEC introduced an
initial suite of ten programs, including six heavy in floating-
point operations. The result of running the ten programs in
this benchmark suite on a processor was captured in a single
“SPECmark” number of merit.
SPECmarks were relative measures of performance based
on a reference machine. For the CPU89 benchmark suite, the
SPEC committee selected the VAX 11/780 as the reference
machine. (The VAX 11/780 was the traditional “1mips” refer-
ence machine used to compute “Dhrystone mips”—see side-
bar on “The Problem With mips.”) By definition, the number
of seconds a VAX 11/780 took to execute each of the ten pro-
grams in the CPU89 suite set the base reference score for each
program. The number of seconds any other machine took to
execute the same program was then divided into its VAX refer-
ence time. This generated a ratio, indicating how much faster
(or slower) than the reference machine the machine being
tested was at executing the given task. The individual ratios for
all ten scores were then averaged to create an overall ratio,
reported as the SPECmark for the tested machine.
For example, in 1989, a 33MHz Motorola 68030 system
achieved a measured SPECmark of 4.0, meaning it was
exactly four times faster than the reference VAX 11/780
machine (on average, across the ten programs in the SPEC
CPU89 benchmark suite). The same year, a MIPS R3000 at
25MHz measured at 17.6 SPECmark. Since both the MIPS
R3000 and the Motorola 68030 were compared with the same
common standard, it also was possible to relate these two very
different types of processors to each other, with the solid con-
clusion that (at least, when executing the set of ten programs
in the SPEC CPU benchmark suite) the R3000 was 17.6/4.0 =
4.4 times faster than the 68030.
In short, for the first time, the SPECmark provided a
level playing field specifically designed to allow meaningful,
MICROPROCESSOR REPORT
6 SPEC CPU2006 Benchmark Suite
direct comparisons of performance across processor architec-
tures. The benchmark suite itself encompassed a reasonable
sample of different, challenging computational tasks. The
same public rules governed running these benchmark pro-
grams and reporting their results. Equally important, it had
been agreed to by all the major players involved in shaping
processor performance. Consequently, the SPECmark
quickly established itself as the single number of merit to use
when comparing processors (replacing all earlier mips-based
performance metrics).
In 1990, the SPEC committee introduced a second
type of performance measurement for multiprocessor sys-
tems, originally termed SPECthruput. Unlike traditional
speed metrics (including SPECmark), SPECthruput is a
measure of throughput performance. This difference is fun-
damental. Speed metrics measure the time it takes to com-
plete a given unit of work—with shorter times awarded
higher marks. (Alternatively, speed can be defined as a
measure of the time it takes to cover a given distance—in a
SPEC benchmark program, specifically, the time it takes to
get from first to last instruction.) By contrast, a throughput
measure computes scores based on the total amount of
work done in a given unit of time (with scores scaling in
direct proportion to the amount of work done, rather than
inversely with the amount of time needed to do the work).
For example, a sports car carrying two passengers
might cover a mile in 18 seconds, achieving a speed mark of
200 miles per hour and a throughput mark of 400 passenger
miles per hour. A bus carrying 60 passengers might cover the
same mile in 60 seconds, achieving a speed mark of 60 miles
per hour and a throughput mark of 3,600 passenger miles
per hour. The point is, a sports car and a bus are fundamen-
tally different types of vehicles and need to be judged by
fundamentally different types of metrics. (A third type of
metric, only now coming into focus for processors—SPEC
expects to introduce its first metric in this area early in
2007—is efficiency. In the present example, efficiency might
be measured by something like the amount of fuel burned
per passenger mile.)
Although the intention behind both SPECmark and
SPECthruput was good, and the basic idea behind each
sound, both measures were reformed and renamed in the
second, 1992, iteration of the SPEC CPU benchmark suite.
The single mix of integer and floating-point programs in
the original benchmark suite was replaced with two separate
suites of programs, one composed of six general integer
programs (with little or no floating-point content) and the
other composed of 14 scientific and technical programs
(heavy in floating-point content).
Corresponding to this bifurcation in the structure of
the SPEC benchmark suite, the single SPECmark measure of
merit was replaced by corresponding separate SPEC Cint92
and Cfp92 measures of merit. Although Cint92 and Cfp92
were both calculated in exactly the same way that SPECmarks
were computed, no method of combining these two scores
© I N - S TAT OCTOBER 10, 2006
into a single number of merit was provided. Instead, the
notion of an overall SPECmark score was simply dropped.
The 1992 SPEC committee also totally revamped the
means of calculating SPECthruput, renaming the reformed
measure SPECrate. Since there were now two separate and
distinct speed measures, one for integer programs and one
for floating-point programs, two correspondingly separate
and distinct integer and floating-point SPECrates were
defined: Cint_rate92 and Cfp_rate92.
The 1995 revision of the SPEC CPU benchmark suite
brought one final refinement in SPEC score reporting. To
counteract the effect of increasingly elaborate (and highly
specific) optimizations being invoked during compilation of
each benchmark program, the 1995 SPEC committee defined
a new required “base” metric for each of the four SPEC CPU
measures, making a total of eight SPEC CPU95 metrics in all.
Although SPEC reporting rules always required disclosure of
all the options set to compile each of the benchmark pro-
grams, the new base metrics imposed strict limits on the
number of compiler flags that could be used and required
that every program in the given suite be compiled with the
same (restricted) set of flags, invoked in the same order.
The idea behind the new restricted base metrics was to
represent the sort of performance ordinary users might
achieve on a tested processor—users who simply wanted
their programs to compile correctly with a minimum of fuss
and were willing to accept whatever performance fell out of
the compiler, processor, and its associated memory hierar-
chy in consequence. The old unrestricted metrics would
continue to represent the sort of performance power users
might achieve on a tested processor—users who were will-
ing to experiment extensively with different combinations
of the available compiler optimizations, in an effort to
wring the last bit of performance out of a program com-
piled to run on their target processor.
Of the two types of measures, SPEC requires vendors to
report only base scores. Reports of peak optimized scores (in
addition) are optional. Although the restrictions placed on
base scores are not foolproof (they can be gotten around, for
example, by building advanced optimizations into the default
settings for a compiler), they are clearly an improvement over
having just peak scores reported. Partly because peak scores
tend to be less consistent, and partly because they tend to be
less representative of “real world” performance, many organ-
izations (including Microprocessor Report) list only base
scores when reporting SPEC CPU numbers. The item that
follows illustrates the sorts of differences found in the num-
ber and complexity of the flags used for base and peak com-
piler optimization, and Table 3 summarizes the changes in
the SPEC benchmark metrics over the years.
IBM Corporation report on IBM System p5 595
(2,300MHz, 1 CPU), August 2006
Base Fortran compiler flags (all Fortran programs):
-O5 -lhmu -blpdata –lmass
MICROPROCESSOR REPORT
 SPEC CPU2006 Benchmark Suite 7

Peak Fortran compiler flags for 178.galgel:
-qpdf1/pdf2
-O5 -qfdpr -qalign=struct=natural -lhmu -blp-
data -lmass -qessl -lessl
fdpr -q -O3
Overall, peak optimization improved the floating-
point scores reported on this machine by just over
8%, but the effect of peak optimization on individ-
ual component scores varied all the way from a
small (less than 1%) negative impact up to an
improvement of nearly 32% (for 178.galgel).
Basic SPEC CPU Speed Arithmetic
SPEC CPU component benchmark programs are never arti-
ficial or made-up programs, designed to simulate or other-
wise represent “typical” workloads. Instead, they are selected
from actual applications used by real people to do meaning-
ful work. Applications are carefully selected to represent dif-
ferent, but (in their own way) equally challenging types of
tasks. It is not necessary that applications be “popular” or
widely used. For the same reason, no attempt is made to
weight applications by frequency of use. The goal of SPEC
CPU is to test processors against demanding workloads
rather than against “typical” workloads. (Assuming what
seems doubtful, that it is possible to determine a “typical”
workload for a “typical” user.)
Although the SPEC measures underwent dramatic
changes in both 1992 and 1995, the basic arithmetic used to
compute processor speed from the run times of the individual
programs in a benchmark suite has not changed since the
original 1989 SPECmark measure itself. Converting run times
for a set of test programs into a test score poses two problems.
The first problem is that, although the SPEC bench-
mark programs are carefully chosen to represent different but
similarly challenging types of tasks, they are nonetheless not
all of the same size and complexity, i.e., some of the test pro-
grams are significantly larger than others, and some have sig-
nificantly longer run times than others. To eliminate this
inequality—giving each test program equal weight (regard-
less of its relative size or complexity)—the run time of each
individual test program is divided into the run time of the
same program on the reference machine, creating a relative
performance ratio. Performance ratios are all the same “size,”
whatever the size and run times of the individual programs
used to determine them.
The second problem is to find a fair way to average the
resulting set of ratios (once every program in the suite has
been run and divided into its corresponding reference
value). With a small set of numbers, obviously mode and
median values are of no significance, while an arithmetical
mean value is liable to be unduly influenced by a single high
or low result. For example, for a suite of ten programs, imag-
ine a test machine twice as fast as the reference machine on

Year Iteration Suites Languages Measures Reference Machine

1989 SPEC CPU Vax 11/780
10 SPEC programs
RT: 18.66 hours C(4) & Fortran(5) & SPECmark 5 MHz
(scores not rounded) C/Fortran(1) SPECthruput 8K cache off-chip
memory N/A
1992 SPEC CPU92 6 CINT92 programs
RT: 6.21 hours C(6) SPECint92
14 CFP92 programs SPECfp92 same as SPEC89
RT: 41.27 hours C(2) & Fortran(12) SPECint_rate92
SPECfp_rate92
(scores rounded to 10s place)
SPECint95
1995 SPEC CPU95
SPECint_base95
8 CINT95 programs SPECfp95 SPARCstation 10/40
RT: 5.25 hours C(8) SPECfp_base95 40 MHz SuperSPARC I
10 CFP95 programs SPECint_rate95 20K/16K I/D L1 on-chip
RT: 11.00 hours Fortran(10) SPECint_rate_base95 no L2 cache
(scores rounded to 100s place) SPECfp_rate95 128MB memory
SPECfp_rate_base95
2000 SPEC CPU2000 12 CINT2000 programs Ultra 5 model 10
C(11) & C++(1)
RT: 5.31 hours 300 MHz UltraSPARC IIi
same set of 8 measures
14 CFP2000 programs 16K/16K I/D L1 on-chip
C(4) & Fortran77(6) & defined for SPEC CPU95
RT: 7.97 hours 2MB L2 cache off-chip
(scores rounded to 100s place) Fortran90(4) 256MB memory
2006 SPEC CPU2006 12 CINT2006 programs C(9) & C++(3) Ultra Enterprise 2
RT: 36.57 hours 296 MHz UltraSPARC II
same set of 8 measures
17 CFP2006 programs C(3) & C++(4) & 16K/16K I/D L1 on-chip
defined for SPEC CPU95
RT: 51.71 hours Fortran(6) & 2MB L2 cache off-chip
(scores not rounded) C/Fortran(4) 1GB memory

Table 3. Changes in SPEC CPU benchmark suites over the years. The benchmark committee stabilized on the current set of eight CPU measures in
1995. The 2000 and 2006 iterations of the benchmark focus on upgrades to the two suites of test programs and refinements of the run and report-
ing rules. The 1992 and 1995 changes in the basic reporting measures themselves were applied retroactively to the former test suites. Thus, for
example, one occasionally sees references to a SPECint89 or a SPECint_base92 score, although no such measures existed in 1989 or 1992.

© I N - S TAT OCTOBER 10, 2006 MICROPROCESSOR REPORT

8 SPEC CPU2006 Benchmark Suite
every component test except one, where it is 100 times as fast.
The arithmetical mean for this set of numbers is 118/10 =
11.8—a value far more misleading than revealing, since the
test machine is not close to being 12 times faster than the ref-
erence machine on any of the test programs.
To solve this problem, the SPEC committee adopted the
geometric mean as the fairest single-value summary of a
small set of numbers, smoothing out the influence of one or
two “extreme” elements. The geometric mean of a set of N
numbers is computed by multiplying the numbers together
and then taking the Nth root of the product. In the present
example, the geometric mean of ten individual SPECratios is
the tenth root of 51,200 = 2.96—a number far more repre-
sentative of the machine’s overall performance than its arith-
metical mean value of 11.8.
Basic SPEC CPU Throughput Arithmetic
SPECrate is SPEC’s “homogeneous capacity method,” mean-
ing it measures a system’s ability to simultaneously execute
separate iterations of the same task rather than of different
tasks. The tasks used for computing SPECrates are the same
SPEC benchmark suites used to compute SPEC speed meas-
ures. SPECrates are measured by simultaneously starting
multiple instances of each program in one of the test suites
on a multiway system and measuring the time needed for all
the copies launched to finish executing. Typically, a separate
copy of the program is started on each separate execution
engine in a system, although there is no SPEC rule requiring
that an eight-way system (for example) start eight instances
of a particular benchmark program. If a vendor thinks a bet-
ter overall result can be achieved by starting instead just seven
instances of a program (reserving one engine to handle sys-
tem overhead), it is free to use an eight-way machine to run
seven-way (or any other-way) SPECrate tests.
Given the number of tasks launched simultaneously,
and the time needed to finish all of them, computing the
number of copies of the given task that could run to comple-
tion on the tested machine in a fixed unit of time, say, one
hour, is straightforward. For example, if an eight-way
machine ran eight copies of a given program in 3,000 sec-
onds, it could run 8 x (3,600/3,000) = 9.6 copies of the pro-
gram in an hour.
But the number of copies of a program that will run to
completion in an hour is a function of the length and com-
plexity of the given program—which can vary greatly. For
example, if eight copies of a second program complete on the
same machine in 1,000 seconds, it could run 8 x (3,600/1,000)
= 28.8 copies of that second program in an hour—not
because the machine suddenly has three times the compute
capacity when executing the second program, but because
executing a copy of the second program involves only one-
third of the work needed to execute the first program.
To determine the compute capacity of a given machine
by running a variety of different programs on it, then,
requires some way to normalize all the different tasks to a
© I N - S TAT OCTOBER 10, 2006
common standard. As with the SPEC CPU speed metrics,
normalization of the throughput measures is accomplished
by relating each individual program in the test suite to the
reference machine. Specifically, an additional normalizing
term (called a “reference factor” by SPEC) is inserted into the
above equation, equal to the execution time (on the reference
machine) of the given benchmark program divided by the
execution time (on the reference machine) of the longest-
running program in the same benchmark suite. In essence,
the reference factor gives the longest-running program in the
suite a weight of 1.0, and every other program in the suite a
fractional weight, in direct proportion to its relative run time
(as determined by the reference machine).
To see how this works, let’s return to the above example,
where we had two very different throughput numbers based
on the very different running times of the two programs used
to measure throughput performance. Using the large 3,000-
second program, the eight-way machine being tested had a
throughput of 9.6 copies an hour. Using the small 1,000-
second program, its throughput was 28.8 copies an hour. Now
suppose these two large and small programs execute on the
reference machine in 9,000 and 3,000 seconds, respectively,
whereas the longest-running program on the reference
machine takes 10,000 seconds. The reference factor for the
first equation, then, is 9,000/10,000, and the reference factor
for the second equation is 3,000/10,000. Inserting these two
terms into the above two throughput calculations then yields:
8 x (9,000/10,000) x (3,600/3,000) = 8.64
8 x (3,000/10,000) x (3,600/1,000) = 8.64
In other words, the reference factor successfully elimi-
nates differences in throughput calculations based on differ-
ences in the size and complexity of the program used to
measure throughput. Of course, in practice, throughput
results obtained by running different programs are unlikely
to be quite as neat as in this particular example. Although the
reference term is an effective way to level out major inequali-
ties between the different programs in a SPEC benchmark
suite, smaller idiosyncratic differences in the way different
programs run on both the test and reference machines are
likely to remain. In consequence, depending on the specific
program used to benchmark throughput, somewhat different
results are likely. Here, just as with the speed metrics, once a
throughput number is determined for each individual pro-
gram in a benchmark suite, an overall throughput number
for the suite is generated by taking the geometric mean of the
individual throughput results.
The overall SPECrate reported for a multiway system,
then, is the geometric mean of the normalized SPECrates of
the component programs in the benchmark suite. This repre-
sents the number of copies of an “average” workload the tested
machine can execute in a given unit of time. When SPECrate
was introduced in 1992, the unit of time used was a week.
SPEC CPU95 dropped the throughput period to a day, and
SPEC CPU2000 further shortened it to an hour. The new
CPU2006 takes an unobvious next step and simply eliminates
MICROPROCESSOR REPORT
 SPEC CPU2006 Benchmark Suite 9

any reference to a fixed unit of time in its SPECrate calcula- The key to comparing machines across different bench-
tion. Specifically, the formula used to compute SPECrates in mark eras is to find a linking machine, i.e., a machine with
the 2006 suite is no longer: scores reported on both the older and newer measures. In a
N x RF x U/T table like Table 4, adjacent machines are always compared on
Where N is the number of copies of the benchmark the same scale (never across different benchmark suites).
program run, RF is the normalizing reference factor for that Linking machines are special only in the fact that they are
benchmark program, U is a specified fixed unit of time (in compared with their predecessor on the older scale and with
seconds), and T is the time (in seconds) the tested machine their successor on the newer scale. As long as each suite fairly
takes to finish all N simultaneously launched copies. Instead, compares the machines of its own era (including the dual-
the revised 2006 formula is simply: membership linking machine), the mathematical law of tran-
N x (RF/T) sitivity allows fair comparisons across successive suites.
Applying this revised formula to the two examples For example, if
above yields: 1. the POWER3-II at 375MHz is fairly regarded as 18.9%
8 x [(9,000/10,000)/3,000] = 0.0024 faster than the PowerPC RS64-III at 450MHz (based on
8 x [(3,000/10,000)/1,000] = 0.0024 their respective SPEC CPU95 benchmark scores of 22.6
In effect, dropping any specific reference to U from vs. 19.0), and
the calculation changes the SPECrate fixed unit of time to 2. the POWER3-II at 450MHz is fairly regarded as 15.3%
the unit used to measure times—in other words, from one faster than the POWER3-II at 375MHz (based on their
hour (3,600 seconds) to one second. (Multiplying 0.0024 x respective SPEC CPU2000 benchmark scores of 286
3,600 = 8.64.) vs. 248). Then it follows (by the law of transitivity) that
3. the POWER3-II at 450MHz (in 2000) is fairly regarded
SPEC CPU As a Window Into the Past as 37.2% faster than the PowerPC RS64-III at 450MHz
SPEC occasionally has been criticized for periodically revising (in 1999)—even though those two machines were never
the programs in its CPU benchmark suite (and changing the scored on a common SPEC benchmark suite.
associated reference machine), on the
grounds that it makes comparisons
from one benchmark-suite era to the
5MHz 5MHz 40MHz 300MHz
next one difficult or impossible. In
VAX VAX SS I US IIi EQV
reality, these changes are not a serious Processor Year MHz INT89 INT92 INT95 INT00 Gain INT00
handicap—as long as one assumes that POWER1 3064 1990 30 24.0 N/A 9.76
each separate suite fairly compares the POWER1 4164 1991 41.67 33.8 38.9% 13.6
systems of its own era. Although SPEC POWER1 6264 1992 62.50 61.3 59.2 50.0% 20.3
POWER2 1993 71.50 125.9 112.7% 43.2
always carefully stresses that no for-
POWER2 1994 71.50 125.9 0.0% 43.2
mula exists for converting measure- PowerPC 604 1995 133 142.2 4.45 12.9% 48.8
ments made using one benchmark PowerPC 604e 1996 200 7.18 61.3% 78.8
suite into measurements for an earlier PowerPC 604e 1997 360 12.1 68.5% 133
or later suite, no such formula is PowerPC 604e 1998 375 14.5 19.8% 159
PowerPC RS64- 1999 450 19.0 31.0% 208
needed to conduct longitudinal stud-
POWER3-II 2000 375 22.6 248 18.9% 248
ies of changes in performance over the POWER3-II 2000 450 286 15.3% 286
years (spanning different SPEC CPU POWER4 2001 1,300 790 176.2% 790
benchmark-suite eras). All that’s nec- POWER4+ 2002 1,450 909 15.1% 909
essary is to find “linking systems” that POWER4+ 2003 1,700 1,077 18.5% 1,077
POWER5 2004 1,900 1,398 29.8% 1,398
connect each scale of reference with
POWER5+ 2005 1,900 1,470 5.2% 1,470
the following scale. POWER5+ 2006 2,300 1,820 23.8% 1,820
IBM’s POWER processor line
furnishes an excellent example for this
sort of study, because its lifespan Table 4. Integer performance of IBM’s POWER/PowerPC processor line from introduction to pres-
ent. As a reminder, the reference machine “scale” for each suite is shown at the top of its column.
(1990–present) coincides almost The POWER3-II at 375MHz is shown in italics because it is just a linking machine and not properly
exactly with the lifespan of the SPEC part of this sequence of best integer performers on a year-by-year basis. It would have been con-
benchmark suite itself (1989–present). venient if IBM had reported a SPEC CPU95 score for 2000’s fastest POWER3-II at 450MHz—directly
Table 4 shows the best SPEC CPU inte- showing its 37% performance advantage over the 450MHz PowerPC RS64-III at 450MHz—but it
ger score achieved by a POWER (or did not. Fortunately, it is not necessary that companies report the most convenient scores, as long as
they report scores for some linking machine, bridging the gap from numbers reported in a former
PowerPC) processor over the 17-year suite to numbers reported in a subsequent suite. (Since SPEC requires that dual scores be reported
history of this architecture, on a year- for a three-month period when transitioning to a new suite, the existence of some crossover machine
by-year basis. or other for every active architecture is virtually guaranteed at each transition point.)

© I N - S TAT OCTOBER 10, 2006 MICROPROCESSOR REPORT

10 SPEC CPU2006 Benchmark Suite

The principle of transitivity also allows machines from was one of great ferment among high-end microprocessor
different SPEC CPU benchmark eras all to be rescaled to any architectures. Intel’s new i860 line threatened the domi-
arbitrary common measure, just as long as the new scale pre- nance of that company’s just-introduced i486 processor
serves the same fair difference between each adjacent pair of family. Motorola’s 68000 line remained an established force,
machines reported on the given SPEC CPU benchmark suite. while the new Motorola 88000 family attempted to super-
Since the base for SPEC CPU2000 was set at 100 rather than sede it. The SPARC, MIPS, and PA-RISC processor architec-
1.00, it makes a convenient general reference scale for all tures contended for RISC preeminence. The DEC VAX,
machines over time (avoiding any need for small fractional Intergraph/Fairchild Clipper, and HP/Apollo Prism were
values all the way down to machines with just 1% the com- still present. The IBM POWER, the DEC Alpha, and the
puting power of a 300MHz UltraSPARC IIi). HP/Intel Itanium had yet to appear.
Note that the integer performance of the POWER/ Today, most of those alternatives are gone. The SPEC
PowerPC line has improved (at least slightly) every year, with the CPU2000 suite serviced just seven architectures: the various
singular exception of 1994, when no advance was posted over x86 lines of Intel and AMD (Pentium, Xeon, Core, Athlon,
the huge (nearly 113%) gain recorded by the new POWER2 Opteron) plus Itanium and the five major high-end RISC
design in 1993. Much can be learned about processor perform- architectures: PA-RISC, MIPS, SPARC, POWER, Alpha. Of
ance from studying a historical table like this one—which, in that diminished pool, at least three architectures (PA-RISC,
turn, is possible only because of the SPEC CPU benchmark suite. MIPS, Alpha) seem certain never to be tested on SPEC
In fact, a table like the one shown in this section, aug- CPU2006. In short, SPEC CPU2006 appears to be a vehicle
mented by similar tables for floating-point performance with a relatively limited practical scope: specifically, com-
and comparable tables for other architectures, is an invalu- paring the various implementations of x64 processors from
able performance tool (albeit, besides POWER, only SPARC Intel and AMD with each other and with alternative SPARC,
and Pentium span the entire period represented by the POWER/PowerPC, and Itanium processors.
SPEC benchmark suite). These sorts of tables provide nearly Even this relatively limited scope appears threatened
endless opportunities for analysis and discussion, backed up with further contraction. As Table 5 shows, exactly one inte-
by hard, measured numbers rather than mere opinion and ger result has been posted (to date) during 2006 for Itanium—
guesswork. Certainly, one of the most significant contribu- despite Intel’s July 18 release of its once much-ballyhooed
tions of the SPEC benchmark suite—empowered by its abil- 1.7-billion-transistor Dual-Core Itanium 2 Processor 9000
ity to periodically refresh itself to stay relevant to the latest series (code-name “Montecito”). Neither the most interest-
generation of processors—is this sort of historical window ing new SPARC processor in years, the UltraSPARC T1
into the changing nature of processor performance itself. (code-name “Niagara”), nor IBM’s recent Cell processor has
bothered to post a single SPEC CPU number.
How Relevant Is SPEC CPU to the Future? To be sure, SPEC CPU has never been a benchmark for
In the 17 years since SPEC CPU89 first appeared, many changes all purposes or all architectures. Embedded processors have
have occurred in the computing industry. The year 1989 their own distinct set of benchmarks, developed by the
Embedded Microprocessor Benchmark
Consortium (EEMBC; for recent coverage,
2000 2001 2002 2003 2004 2005 2006 Total see MPR 2/22/05-01, “EEMBC Expands
Xeon 0 9 16 29 45 118 130 347
Pentium 36 29 62 31 41 37 35 271
Benchmarks,” and MPR 7/17/06-02,
Opteron 17 14 68 74 173 “EEMBC Energizes Benchmarking”). No
SPARC 6 43 19 19 7 8 8 110 IBM mainframe or mainframe clone has
POWER 16 13 18 21 10 10 21 109 ever been subjected to SPEC CPU testing,
Itanium 4 7 15 21 10 1 58 nor has any Cray or other supercomputer.
Athlon 2 14 11 6 4 2 6 45
Alpha 15 8 10 3 4 40 And even though the candidate micro-
Core 23 23 processor architectures for SPEC CPU test-
PA-RISC 5 8 4 17 ing are dwindling down to a select few, the
R1X00 (MIPS) 5 4 2 11 number of vendors anxious to test their sys-
Total Cint2000 reports 85 132 149 141 146 253 298 1,204
tems on SPEC CPU measures seems, if any-
Table 5. Breakdown of SPEC CPUint2000 results by processor type. Although 2002 was the last
thing, to be on the increase (see Table 6).
year any results were reported for either PA-RISC or MIPS processors, and 2004 marked the last
report for an Alpha processor, architectural dropout has not cast a damper on results reporting. A Brave New World of
To the contrary, reporting has picked up sharply in the past two years as a consequence of the Processor Performance
ongoing performance duel between Xeon-based and Opteron-based systems. New Core-based A more serious difficulty than the dwindling
systems may fuel a further increase in reporting during 2007. The sharp drop-off in reported
results for SPARC-based systems, starting in 2004, reflects the fact that Sun has declined to post
number of high-end architectures targeted
any Cint2000 results for its throughput-oriented UltraSPARC IV generation, leaving Fujitsu to by the SPEC CPU benchmark suite is the
carry the burden of single-thread performance with its alternative SPARC64 V line. changing nature of high-end processors

© I N - S TAT OCTOBER 10, 2006 MICROPROCESSOR REPORT

 SPEC CPU2006 Benchmark Suite 11

themselves. The introduction of both multicore chip designs

(starting with IBM’s POWER4, introduced in 2001) and mul-
Price & Availability
tithreaded core designs (starting with Intel’s “hyperthreaded”
Pentium 4 Xeon, introduced in 2002) has changed the very SPEC CPU2006 is available for immediate delivery (on
notion of processor performance. DVD) from SPEC. Since SPEC is a nonprofit corporation,
With a widespread shift to dual-core designs already in the price is modest, intended only to defray associated
full swing, and still higher core counts promised for following costs. For new customers, the charge is $800; for cus-
generations, traditional single-core, single-thread processors tomers upgrading from SPEC CPU2000, the cost is $400;
are rapidly becoming extinct. In a multicore, multithread and for qualified educational institutions, the cost is just
milieu, every processor is (or soon will be) a multiprocessor $200. Requirements for running the benchmarks on a test
system on a chip. For these designs, performance evaluation system include a Windows, Unix, or Linux operating sys-
must inevitable shift away from the traditional speed metrics tem plus a minimum of 1GB of memory, at least 8GB of
in the SPEC CPU benchmark suite toward throughput meas- unused disk space, and a set of C, C++, and Fortran com-
ures. Indeed, some of the newest designs, like Sun’s Ultra- pilers (plus, of course, a DVD drive to read the SPEC disks).
SPARC T1 (8 integer cores x 4 threads/core) and IBM’s Cell See www.spec.org/ for additional details about the
broadband engine (1 dual-thread PowerPC core + 8 vector SPEC benchmark suites.
synergistic processor elements or SPEs) already stress the
throughput half of the speed/throughput dichotomy much
more heavily than any dual-core and/or dual-thread design. SPEC measures, it must be that SPEC measures do not fairly or
Regrettably, the SPECrate measure, although useful, is usefully reflect the performance these sorts of radical multi-
both incomplete and inadequate as a sole measure of through- thread designs provide.
put performance. There are several issues here. Vendors that
want to post a throughput score but not a speed score For 2006, Just a New CPU Suite Is Not Enough
sometimes hesitate to use SPECrate, because of its link to the The continued widespread use of the SPEC CPU bench-
SPEC speed metrics (fearing that competitors may attempt mark suite to exhibit and compare processor performance is
to reverse engineer a speed number out of a posted a testament to the admirable job done by the SPEC CPU
SPECrate number). Set-
ting this issue aside,
SPECrate measures only 4Q99 2000 2001 2002 2003 2004 2005 1-3Q06 Total
“homogeneous” work- 1 Acer Incorporated 0 0 0 0 0 0 25 4 29
loads, but many multi- 2 Advanced Micro Devices 0 2 14 11 19 9 17 10 82
3 Alpha Processor, Inc. 0 2 0 0 0 0 0 0 2
threaded workloads are 4 ASUS Computer International 0 0 0 0 0 0 0 6 6
heterogeneous and so 5 Banha for Electronic Industeries 0 0 0 0 0 0 0 1 1
remain unrepresented by 6 Bull 0 0 0 0 3 1 0 10 14
any existing SPEC bench- 7 CAIRA 0 0 0 0 0 0 0 2 2
8 Compaq Computer Corporation 6 7 9 1 0 0 0 0 23
mark number. Further- 9 Dell 2 11 19 39 28 22 21 38 180
more, SPECrate was ini- 10 Fujitsu* 0 1 28 12 10 7 30 31 119
tially conceived and still 11 gcocco software, inc 0 0 1 0 0 0 0 0 1
13 Hewlett-Packard Company 0 5 11 24 22 26 46 45 179
exists primarily as a metric
12 Hitachi 0 0 0 0 0 0 3 0 3
for comparing multi- 14 IBM Corporation 1 15 13 18 23 14 68 84 236
processor systems, rather 16 Intel Corporation 1 21 18 33 17 22 3 22 137
than for comparing indi- 15 ION Computer Systems 0 0 0 0 1 10 7 15 33
17 PathScale, Inc. 0 0 0 0 0 1 0 0 1
vidual processors at the 18 RLX 0 0 0 2 0 0 0 0 2
level of multiprocessor sys- 19 SGI 2 3 4 2 4 7 1 0 23
tems on a chip. 20 Sun Microsystems 1 5 13 7 14 7 9 11 67
It is significant that 21 Supermicro 0 0 0 0 0 20 23 19 62
22 Verlag Heinz Heise 0 0 2 0 0 0 0 0 2
neither of the two most Total Cint2000 reports 13 72 132 149 141 146 253 298 1,204
interesting recent processor Total companies reporting 6 10 11 10 10 12 12 14
designs—Sun’s Ultra-
SPARC T1 and IBM’s Cell Table 6. Breakdown of companies reporting SPEC CPUint2000 results. The increase in Xeon and Opteron report-
processor—are represented ing during 2005–2006 (shown in Table 5) has been largely driven by Acer, Bull, Dell, Fujitsu, HP, IBM, Sun, and
anywhere in the current set Supermicro. (All Sun Cint2000 results from 2004 on are for Opteron systems.) The 14 different companies report-
ing results in 2006 (to date) is also an all-time high. In addition to the 22 companies listed above, six more com-
of posted SPEC CPU panies have reported results in one or more of the other three SPEC CPU results categories (for floating-point or
results. Since neither Sun rate numbers): HCL Infosystems, Itautec, NEC, RackSaver, Unisys, and Wipro. *Includes results from Fujitsu Lim-
nor IBM hesitates to use ited (41), Fujitsu Siemens Computers (77), and Fujitsu Egypt (1).

© I N - S TAT OCTOBER 10, 2006 MICROPROCESSOR REPORT

12 SPEC CPU2006 Benchmark Suite
committee over the years. The efforts of this committee to
The Problem With “mips”
SPEC was born out of a controversy over processor per-
formance that raged during the mid-to-late 1980s, at the
time of the “RISC revolution” in processor design. Before
the introduction of the SPEC benchmark suite, processor
performance was measured in “mips,” an acronym for mil-
lions of instructions executed by a processor per second.
(Note that mips is not a plural form; 1mips is the proper
performance designation for a machine able to execute one
million instructions per second.) There is nothing intrinsi-
cally wrong with measuring performance by the number of
instructions executed in a unit of time. To the contrary,
computer architects typically spend their time thinking of
ways to get a next-generation machine to execute more
instructions per second than the current generation of
machine. Thus, the waggish translation of mips as “mean-
ingless indicator of processor speed”—popular during the
height of the RISC performance wars—was unduly harsh.
The problem with mips as a measure of processor per-
formance is not that it’s meaningless, but rather that its
meaning is not fixed. There are peak mips, which is the
maximum number of instructions a particular machine could
possibly execute in a second. This can be computed by sim-
ply multiplying the number of instructions a machine can
issue in a clock cycle by its clock speed in megahertz. Thus,
the Intel i860, one of the early superscalar designs, was pro-
moted as offering up to “150mips” of performance, based
on its (intended) clock speed of 50MHz and its ability to
issue a maximum of three instructions in a single clock cycle.
A peak mips rating, however, gives a very unrealistic
appraisal of a machine’s actual performance. A better
measure is “sustainable mips,” which is computed by mul-
tiplying a processor’s clock speed by the average number of
instructions it issues in a clock cycle during the course of
some reasonable workload. Thus, the initial SPARC-based
systems from Sun were advertised as “10mips” machines.
Since the processor ran at 16.67MHz, clearly somebody at
Sun estimated a typical issue rate of about six instructions
out of every 10 clock cycles, or 60% of its peak issue rate
of one instruction per clock.
Peak vs. sustainable mips ratings had opposite virtues
and vices. Peak mips were clear, definite values that every-
one could agree on, but they were very unrealistic in terms
of a machine’s achieved (or even achievable) performance.
Average mips might more closely reflect the machine’s
achieved performance, but there was nothing clear or def-
inite about them. Not only did they vary quite dramatically
by workload, but the manner of computing an average
number of instructions per clock was often only vaguely
specified, if it was specified at all.
© I N - S TAT OCTOBER 10, 2006
periodically refresh and renew the SPEC CPU benchmark
The biggest difficulty with both peak and sustainable
mips ratings, however, is that comparisons are properly lim-
ited to machines that share a common ISA. That is to say,
it makes sense for a computer architect to use mips when
thinking about the difference in performance between two
generations of machines based on the same ISA. But where
ISAs differ, the same mips number will mean two different
things. Particularly when ISA differences are as profound as
RISC vs. CISC, mips comparisons are likely to be very mis-
leading. CISC proponents rightly complained that their
“complex” instructions tended to do more work than the
“reduced” instructions RISC machines executed, There-
fore, CISC mips were bigger than RISC mips, e.g., a
“3mips” CISC machine might actually outperform a
“5mips” RISC machine (i.e., do more work in three million
instructions than a given RISC machine would accomplish
by executing five million instructions).
These sorts of intractable difficulties with peak and
sustainable mips led to “Dhrystone mips” ratings (also
known as “VAX mips”). Dhrystone mips were not mips at
all, except by courtesy. This value was computed by count-
ing the number of times a small artificial integer-code
benchmark program called “Dhrystones” (a pun on a well-
known floating-point benchmark known as “Whetstones”)
iterated in a second and dividing that number by 1,760—
supposedly, the number of times this program iterated in a
second on a VAX 11/780 (a 1978 DEC system widely pro-
moted as a “1mips” machine, despite the fact that it exe-
cuted only about 500,000 instructions a second).
Dhrystone mips enjoyed several advantages. It was
both clear and measurable (in contrast to sustainable mips),
bore some relationship to actual performance (in contrast
to peak mips), and could be used to compare machines
with very different ISAs (in contrast to either peak or sus-
tained mips). It suffered from two disadvantages: its only
connection to mips was an exaggerated DEC marketing
claim and the awkward truth that the artificial benchmark
itself was not representative of any realistic application pro-
gram (see MPR 12/16/02-01, “Dhrystones Are All Wet”).
Still, the notion of Dhrystone mips was clearly a step
in the right direction. The basic idea itself—compare the
performance of a benchmarked machine against the
known performance of a standard reference machine on a
standardized test program—was exactly right. The problem
lay in using Dhrystone as the standardized test program. It
was simply too small and too prone to compiler optimiza-
tion to be a reliable measure. The solution adopted by SPEC
was to replace this one small, artificial program with a
robust suite of much larger programs, drawn from real life
MICROPROCESSOR REPORT

P R O B L E M W I T H “ M I P S ” (C o n t i n u e d )
and carefully chosen to represent a range of important
and typical computational tasks. As a side benefit,
because machine speed was directly measured by the
elapsed time needed to complete specific programs
rather than by counting the number of instructions exe-
cuted per second, the notion of “mips” could be
dropped altogether in the various SPEC measures.
suite have kept it current with, and relevant to, successive
generations of processors. However, if SPEC CPU2006 is to
remain the computer industry’s premier benchmarking tool
for analyzing and comparing processor performance over
the next five years, more than just a revision of the pro-
grams in the CPU2000 benchmark suite may be needed. For
the first time in over a decade, it may be necessary for the
SPEC committee to revisit the basic scales created for meas-
uring processor performance.
In particular, it may be time to broaden the scope of
the measures comprehended in SPEC CPU2006 beyond the
current three dichotomies of integer vs. floating-point,
speed vs. rate, and base vs. peak performance. Without
question, the focus of SPEC CPU2006 should remain on
component processor performance, but 2007 should see an
expansion in the focus of this component to encompass all
the relevant performance issues raised by current and future
radically multithreaded processor chips.
New measures should be in addition to the current,
older measures. Certainly, Cint2006 and Cfp2006 will con-
tinue to adequately address the sort of performance
achieved by a single core running a single thread. As IBM’s
next-generation POWER6 design indicates, this basic speed
To subscribe to Microprocessor Report, phone 480.483.4441 or visit www.MPRonline.com
© I N - S TAT OCTOBER 10, 2006
SPEC CPU2006 Benchmark Suite 13
metric will never cease to be relevant and interesting. Use of
the current “auto-parallelization” option, when compiling
and running the individual programs in the two speed
benchmark suites, allows the multiple cores provided by
today’s processors to be applied to the task of speeding up a
single thread of execution. (Although it may be desirable to
revisit the current single-thread measures—with and with-
out auto-parallelization—looking for potential weaknesses
and shortcomings in the light of new, massively threaded
chips like the UltraSPARC T1 and the IBM Cell.)
Current measures of the throughput performance a
single processor chip can deliver are less satisfactory, how-
ever. CPU2006 SPECrate numbers provide only limited illu-
mination here, in part because this measure was never
designed to address performance at the component proces-
sor level (but only at the multiprocessor system level).
Another vital issue for the processors of 2006 and
beyond is the efficiency with which a processor chip deliv-
ers performance, both from the standpoint of its speed rat-
ing and in terms of its throughput rating. Is there, say, an
optimal speed and/or throughput performance-per-watt
figure for a processor, and, if so, how can a customer know
when this ideal balance point is reached?
The SPEC committee is already hard at work develop-
ing ways to measure a processor’s efficiency (due to be
released sometime in early 2007). This is a positive sign that
SPEC intends to continue changing with the times in the
future, as it has in the past. It is to be hoped that the SPEC
committee will not rest after adding new efficiency meas-
ures. The processors available for testing by the CPU2006
benchmark suite will be more different from the processors
tested on CPU2000 than CPU2000 processors were from
the processors tested by the original CPU89 suite. It is time
to revisit the fundamental notions of measuring processor
performance.
MICROPROCESSOR REPORT