Limitations of Serial Computing:

1) Physical and practical constraints to simply build ever faster serial

computers.
2) Transmission Speeds: How fast the data can move through hardware and that
is restricted by the hardware itself.
3) Economic: Expensive to make a single processor faster, using more numbers
of same processor to achieve the same or better performance is inexpensive.

Serial Computing:
1) Problem broken into discrete series of instructions.
2) Instructions execute one after another.
3) Only one instruction may execute at any moment in time.

Parallel Computing:
1) Problem is broken into discrete parts that can be solved concurrently.
2) Each part is then broken into series of instructions.
3) Instructions of each part can execute simaltaneously on different CPUs.

Why Parallel Computing:

1) Save time
2) Solve larger problem
3) Provide parallelism

Flynn Taxonomy: Classification of Computer Architectures according to Instructions

and Data. (Single or Multiple)

Granularity: Ratio of computation to communication

1) Coarse: large amount of computational work between communication events.
(more work, less communication)
2) Fine: small amounts of computation work between communication events.
(less work, more communication)

Observed Speedup: Speedup of parallelize code: serial time / parallel time

Parallel Overhead:
*) Task start-up time
*) Task termination time
*) Synchronization
*) Data Communication
*) Software Overhead

Massively Parallel: Hardware with parallel system (having many processors).

Scalability: Systems ability to demonstrate a proportionate increase in parallel

speedup with addition of more processors.

Factors that contribute to scalability:

*) Hardware:
*) Application Algorithm
*) Parallel overhead
*) Characterestics of specific app and coding

Shared Memory: Multiple CPU and a common shared memory hence cache coherence is
also there
1) UMA = One SMP, identical processors with equal access and access times to
memory
2) NUMA = Generally two or more SMP (unequal memory size + access times)
Advantages:
*) Global address space, programmer friendly.
*) Data sharing between tasks is uniform and fast.
Disadvantages:
*) Lack of scalability (adding more cpu congests the shared memory-cpu path)
+ phyiscal limitation
*) Programmer responsibility for synchronization (tasks accessing global
memory)
*) Expensive to build machines with ever increasing numbers of processors

Distributed: Different systems connected via network, no global memory, each

processor with a local memory hence no cache coherence.

Advantages:
*) Scalable (processors and memory consequently)
*) Processors can quickly access local memory without interference/overhead
because no cache coherence is to be maintained.
*) Cost Effective as any normal system can be used/added

Disadvantages:
*) Programmer responsibility for setting up data communication between
processors
*) Existing data structures, based on global memory, can't be mapped to this
memory organization.
*) Memory access time is non uniform.