Lecture 2: Performance Metrics

• Today’s topics:

 Performance Metrics
 Execution Time
 Evaluating a system

Yosef A. Abdulmoghni Al-Razi University 1

Performance Metrics
•Performance is one of the main factors in chosen
between computers
Important for both marketing and purchasers

•Modern computer systems performance assessment is

very difficult.
 The Intricacy of software systems
 The wide range of performance improvements
employed by HW designers
• We aim at:
 Why a software systems performs as it does
 How can an instruction be implemented to run
better than another
 How hardware components affect the performance
Yosef A. Abdulmoghni Al-Razi University 2
Performance Metrics

• Possible measures
 response/execution time – time elapsed between
start and end of a program
 throughput – amount of work done in a fixed time

• The two measures are usually linked

 A faster processor will improve both
 More processors will likely only improve throughput

• What influences performance?

Yosef A. Abdulmoghni Al-Razi University 3

Execution Time

Consider a system X executing a fixed workload W

PerformanceX = 1 / Execution timeX

Execution time = response = wall clock = elapsed time=

time to execute the workload + Memory access + Disk
Access + IO as well as time spent by the operating system
coordinating various events

The are utilities to break up the wall clock time as user and
system time

Yosef A. Abdulmoghni Al-Razi University 4

Speedup and Improvement

• System X executes a program in 10 seconds, system Y

executes the same program in 15 seconds

• System X is 1.5 times faster than system Y

• The speedup of system X over system Y is 1.5 (the ratio)

• The performance improvement of X over Y is

1.5 -1 = 0.5

• The execution time reduction for the program, compared to

Y is (15-10) / 15 = 33%
The execution time increase, compared to X is
(15-10) / 10 = 50%
Yosef A. Abdulmoghni Al-Razi University 5
Speedup and Improvement

• CPU time is the time the CPU spends working on our

behalf
• does not include time executing other tasks or waiting for
IO
• Includes
 User CPU time. Time spent on the program
 System CPU time: Time spent on OS executing tasks
on behalf of the program
• System performance refers to elapsed time on unloaded
system
• CPU performance refers to user CPU time
• Clock cycles or ticks
• Clock period time to finish a cycle
• Clock rate or frequency is the inverse of the period time
Yosef A. Abdulmoghni Al-Razi University 6
Performance Equation - I

CPU execution time = CPU clock cycles x Clock cycle time

Clock cycle time = 1 / Clock rate

If a processor has a frequency of 3 GHz, the clock ticks

3 billion times in a second – as we’ll soon see, with each
clock tick, one or more/less instructions may complete

If a program runs for 10 seconds on a 3 GHz processor,

how many clock cycles did it run for?

If a program runs for 2 billion clock cycles on a 1.5 GHz

processor, what is the execution time in seconds?

Yosef A. Abdulmoghni Al-Razi University 7

Example

A program run 10s on ComputerA which has a 4GHz clock

We need to design a ComputerB to run the same program
in 6 seconds. We can increase the Clock rate but that will
affect the internal design so that the program takes 1.2 more
cycles than it used to on CopmputerA. What is the target
Clock rate for Computer B?

Yosef A. Abdulmoghni Al-Razi University 8

 Performance Equation - II

CPU clock cycles = number of instrs x avg clock cycles

per instruction (CPI)
Substituting in previous equation,

Execution time = clock cycle time x number of instrs x avg CPI

• If a 2 GHz processor graduates an instruction every third

cycle, how many instructions are there in a program that
runs for 10 seconds?
• Two Computer Implement the same ISA! Which is faster?
CompA cycle time is 250ps and Average CPI=2 for prog1
CompB cycle time is 500ps and Average CPI=1.2 for prog1
Yosef A. Abdulmoghni Al-Razi University 9
 Factors Influencing Performance

Execution time = clock cycle time x IC x avg CPI

• Clock cycle time: manufacturing process (how fast is each

transistor), how much work gets done in each pipeline stage
(more on this later)

• Instruction Count (IC) or Number of instrs: the quality of the

compiler and the instruction set architecture

• CPI: the nature of each instruction and the quality of the

architecture implementation

Yosef A. Abdulmoghni Al-Razi University 10

 Example

Execution time = IC x CPIavg x CT

Which of the following two systems is better?

• A program is converted into 4 billion MIPS instructions by a

compiler ; the MIPS processor is implemented such that
each instruction completes in an average of 1.5 cycles and
the clock speed is 1 GHz

• The same program is converted into 2 billion x86 instructions;

the x86 processor is implemented such that each instruction
completes in an average of 6 cycles and the clock speed is
1.5 GHz
Yosef A. Abdulmoghni Al-Razi University 11
Example

• Two programs running on the same computer

CPI for Instruction Class

A B C
CPI 1 2 3

IC for each Prog

A B C
Prog A 2 1 2
Prog B 4 1 1

• Which Program execute more instructions?

• Which is faster?
• What is the average CPI for each?
Yosef A. Abdulmoghni Al-Razi University 12
Benchmark Suites

• Measuring performance components is difficult for most

users: average CPI requires simulation/hardware counters,
instruction count requires profiling tools/hardware counters,
OS interference is hard to quantify, etc.

• Each vendor announces a SPEC rating for their system

 a measure of execution time for a fixed collection of
programs
 is a function of a specific CPU, memory system, IO
system, operating system, compiler
 enables easy comparison of different systems

The key is coming up with a collection of relevant programs

Yosef A. Abdulmoghni Al-Razi University 13
SPEC CPU

• SPEC: System Performance Evaluation Corporation, an industry

consortium that creates a collection of relevant programs

• The 2006 version includes 12 integer and 17 floating-point applications

• The SPEC rating specifies how much faster a system is, compared to
a baseline machine – a system with SPEC rating 600 is 1.5 times
faster than a system with SPEC rating 400

• Note that this rating incorporates the behavior of all 29 programs – this
may not necessarily predict performance for your favorite program!

Yosef A. Abdulmoghni Al-Razi University 14

Deriving a Single Performance Number

How is the performance of 29 different apps compressed

into a single performance number?

• SPEC uses geometric mean (GM) – the execution time

of each program is multiplied and the Nth root is derived

• Another popular metric is arithmetic mean (AM) – the

average of each program’s execution time

• Weighted arithmetic mean – the execution times of some

programs are weighted to balance priorities

Yosef A. Abdulmoghni Al-Razi University 15

Amdahl’s Law

Yosef A. Abdulmoghni Al-Razi University 16

Amdahl’s Law

• A program runs on a computer for 100s with 80% of its time

on CPU and 20% in I/O. How much do I have to improve the
speed of the CPU part if I want the program to run twice as
fast?

Yosef A. Abdulmoghni Al-Razi University 17

MIPS

• Million Instructions Per Second

MIPS = IC/(ET x 106)
• Issues
1. Cannot compare two computers with different ISA
coz IC will differ
2. A single Computer has a different MIPS for each
program
• Use MIPS and ET to determine which compiler is faster
on a 1 GHz processor
Code IC in Billions
CPI for this instruction Class Sequence A B C
A B C Comp 1 5 1 1
CPI 1 2 3 Comp 2 10 1 1

Yosef A. Abdulmoghni Al-Razi University 18

Closing Remarks

Stick to Execution Time

Yosef A. Abdulmoghni Al-Razi University 19