Distributed-memory

parallel programming
with MPI

Supervised by: Dr. Shaima Hagras

 Agenda

1.

2.

3.
Introduction
Topic one
What is MPI ?
• Standard for Message Passing Interface
• is a standardized interface for exchanging messages between multiple computers running a
parallel program across distributed memory.
• The MPI standard defines the syntax and semantics of library routines that are useful to a wide
range of users writing portable message-passing programs in C, C++, and Fortran.
• MPI supports both point to point and collective communication between processes, as well as
derived data types, one sided communication, dynamic process management, and I/O.
• MPI is widely used for developing and running parallel applications on the Windows platform, as
well as on Linux and other operating system.

Com_1 Com_2

Com_3 Com_4
processes are the actual instances of the program
that are running.

MPI allows you to create logical groups of processes, and

in each group, a process is identified by its rank.
This is an integer in the range [0, N-1] where N is the size
of the group.

Communicators are objects that handle communication

between processes.
MPI uses communicators to organize how processes
communicate with each other.

An intra-communicator handles processes within a

single group,
while an inter-communicator handles communication
between two distinct groups.

Note that process ranks are relative to a

communicator. A program may have multiple
communicator; if so, a process may have multiple
ranks, one for each communicator it is associated with
 copy copy copy

rank = 0 rank = 1 rank = 2

What is the goal of MPI ?
The goal of MPI is to establish a portable , efficient, and flexible
standard for message passing that will be widely used for
writing message passing programs

Why MPI ?

1. standard framework for distributed computing.

2. simple communication model between processes in a program.
3. Enforces certain guarantees:
• Reliable message
• In-order arrival of messages
4. multiple implementations and high efficient for different
platform.
5. Designed for multi node technical computing
Topic two
 point-to-point communication
MPI Point-to-Point communication is the most used
communication method in MPI.

It involves the transfer of a message from one process to a

particular process in the same communicator.

MPI provides blocking (synchronous) and non-blocking

(asynchronous) Point-to-Point communication.

With blocking communication, an MPI process sends

a message to another MPI process and waits until the
receiving process completely and correctly receives
the message before it continues its work.

On the other hand, a sending process using non-

blocking communication sends a message to another
MPI process and continues its work without waiting to
ensure that the message has been correctly received
by the receiving process.
If there is exactly one sender and one receiver we
speak of point-to-point communication. Both ends
are identified uniquely by their ranks.

Each point-to-point message can carry an

additional integer label, the so-called tag, which
may be used to identify the type of a message,
and which must match on both ends.

Tag: Arbitrary non-negative integer assigned by the

programmer to uniquely identify a message. Send
and receive operations should match message tags.
Topic three
 Collective communication in MPI

1. MPI, too, has mechanisms that make reductions much simpler and in most
cases more efficient than looping over all ranks and collecting results.
2. Collective communication, as opposed to point-to-point communication.
3. collective communication is that it implies a synchronization point among
processes. This means that all processes must reach a point in their code
before they can all begin executing again.

synchronization point among processes :

The barrier synchronizes the members of the

communicator, i.e., all processes must call it
before they are allowed to return to the user
code MPI_Barrier is the simplest collective in
MPI, it does not actually perform any real data
transfer.
MPI Broadcast :
A broadcast is one of the standard collective communication
techniques. During a broadcast, one process sends the same data
to all processes in a communicator. One of the main uses of
broadcasting is to send out user input to a parallel program, or
send out configuration parameters to all processes.

An MPI broadcast: The “root” process (rank 1 in this example) sends the same message to all
others. Every rank in the communicator must call MPI_Bcast() with the same root argument.
 Data distribution :
Data distribution in MPI refers to how data is distributed among processes in a parallel computing
environment that uses the Message Passing Interface (MPI).
MPI allows processes to communicate and exchange data in a distributed-memory system. Proper
data distribution is crucial for load balancing and efficient parallel execution of algorithms.

common strategies for data distribution in MPI:

 Examples of more advanced MPI collective calls
that are concerned with global data distribution:
1-MPI_Gather() collects the send buffer contents of all processes and concatenates them in
rank order into the receive buffer of the root process
2- MPI_Scatter() does the reverse, distributing equal-sized chunks of the root’s send buffer
3- MPI_Allgather() is a combination of MPI_Gather() and MPI_Bcast().

MPI_Scatter

MPI_Scatter is a collective routine that is

very similar
MPI_Scatter involves a designated root
process sending data to all processes in
a communicator

The primary difference between MPI_Bcast and MPI_Scatter is small but

important. MPI_Bcast sends the same piece of data to all processes
while MPI_Scatter sends chunks of an array to different processes.
MPI_Bcast takes a single data element at the root process (the red box) and
copies it to all other processes.
MPI_Scatter takes an array of elements and distributes the elements in the order of process
rank. The first element (in red) goes to process zero, the second element (in green) goes to
process one, and so on.

Although the root process (process zero) contains the entire array of data,
MPI_Scatter will copy the appropriate element into the receiving buffer of
the process. Here is what the function prototype of MPI_Scatter looks like.
 MPI_Gather :

Similar to MPI_Scatter, MPI_Gather takes

elements from each process and gathers them
to the root process. The elements are ordered
by the rank of the process from which they
were received. The function prototype for
MPI_Gather is identical to that of MPI_Scatter

In MPI_Gather, only the root process needs to have a

valid receive buffer. All other calling processes can
pass NULL for recv_data. Also, don’t forget that the
recv_count parameter is the count of elements
received per process, not the total summation of
counts from all processes. This can often confuse
beginning MPI programmers.
MPI_Allgather

Given a set of elements distributed across all

processes, MPI_Allgather will gather all of the
elements to all the processes. In the most basic
sense, MPI_Allgather is an MPI_Gather
followed by an MPI_Bcast. The illustration
below shows how data is distributed after a call
to MPI_Allgather.
Thank you