Parallel Computing Landscape

(CS 526)

Muhammad Yasir,

Department of Computer Science,

The University of Lahore,
 Serial Performance Scaling is Over
• Cannot continue to scale processor frequencies
– no 10 GHz chips

• Cannot continue to increase power

consumption
– can’t melt chip

• Can continue to increase

transistor density
 The “New” Moore’s
Law
• Computers no longer get faster, just wider

• You must re-think your algorithms to be

parallel !

• Data-parallel computing is most scalable

solution:

2 cores 4 cores 8 cores 16 cores…

 Generic Multicore
Chip
Processor Local Processor Local
Memor Memor
y y

Global Memory

• Handful of processors each supporting ~1 hardware threads

• On-chip memory near processors (cache, RAM, or both)

• Shared global memory space (external DRAM)

 Generic Many-core Chip
Processor Memory Processor Memory
•••

Global Memory

• Many processors each supporting many hardware threads

• On-chip memory near processors (cache, RAM, or

both)

• Shared global memory space (external DRAM)

 Emergence of Parallel Architectures
– Multi-core processors:
• Processors having n computing cores
Multi-cores
- Transistors

- Clock Speeds

- Power

- Performance
(Perf/Clock) ILP
 The memory hierarchy
• If simultaneous multithreading only:
– all caches shared

• Multi-core chips:
– L1 caches private
– L2 caches private in some architectures
and shared in others

• Memory is always shared

 Multi-cores – Memory Hierarchies
hyper-threads
• Dual-core
Intel Xeon processors

CORE1

CORE0
• Each core is L1 cache L1 cache
hyper-threaded (SMT)
L2 cache

• Private L1 caches
memory
• Shared L2 caches
 Designs with private L2
caches

CORE1

CORE0

CORE1

CORE0
L1 cache L1 cache L1 cache L1 cache

L2 cache L2 cache L2 cache L2 cache

L3 cache L3 cache
memory
memory
Both L1 and L2 are private
Examples: AMD Opteron,
A design with L3 caches
AMD Athlon, Intel Pentium D
Example: Intel Itanium 2
 Private vs Shared caches?

• Advantages ???
 Private vs Shared caches
• Advantages of private:
– They are closer to core, so faster access
– Reduces contention

• Advantages of shared:
– Threads on different cores can share the same cache
data
– More cache space available if a single (or a few) high-
performance thread runs on the system