Wolkite University

College of Computing and Informatics

Information Technology Department

Introduction to Distributed System

ITec4102

Chapter 6: Synchronization
March 18, 2020

Fuad Yimer
 Overview

Overview

We Discuss
The issue of synchronization based on time (actual time and relative
ordering)
How a group of processes can appoint a process as a coordinator; can be
done by means of election algorithms
Distributed mutual exclusion to protect shared resources from
simultaneous access by multiple processes
Distributed transactions, which also do the same thing, but optimize
access through advanced concurrency control mechanisms

2 / 38
 Overview Essentiality of Synchronization

Essentiality of Synchronization

Synchronization is all about doing the right thing at the right time.
Communication between processes requires total ordering of all actions.
Total ordering of events
Total ordering of execution
Ordering of access to resources
Thus, synchronization of all actions forms the heart of distributed
systems, and requires some concept of time.
In distributed systems, synchronization is a necessary part of cooperation
and coordination between processes.

3 / 38
 Overview Issues in Synchronization

Issues in Synchronization

Inherent limitations of a distributed system

Absence of global clock
- difficult to make temporal order of events
- difficult to collect up-to-date information on the state of the entire system
Absence of shared memory
- no up-to-date state of the entire system to any individual process as there’s
no shared memory
Time and Clocks
Synchronizing clocks and using time in distributed algorithms
Global State
how to acquire knowledge of the system’s global state
Concurrency Control
coordinating concurrent access to resources
Coordination
when do processes need to coordinate and how do they do it
4 / 38
 Clock Synchronization

Clock Synchronization

Physical clocks
Clock Synchronization
Basics
Algorithms

5 / 38
 Clock Synchronization Physical clocks

Physical clocks
Sometimes we need the exact time to timestamp events in distributed
system.
But what exactly is a second, or a day?
The earth is slowing down due to tidal friction and atmospheric drag.
The day is becoming longer.
Mean solar second is computed as [avg(largeNumberofdays)]/86400.
How do we measure time?
’Absolute’ time [Einstein says no absolute time]
Astronomical time
- Based on earth’s rotation
- Not stable as the length of earths rotation varies.
- E.g., Greenwich Mean Time (GMT)
International Atomic Time (IAT)
- Based on oscillations of Cesium-133 atom.
- But a day in IAT is 3 msec less than a mean solar day.
Coordinated Universal Time (UTC)
- Calculated as an average of several worldwide atomic clocks.
- Leap seconds adjustment is made to compensate the discrepancy between
IATs day and mean solar day.
6 / 38
 Clock Synchronization Clock Synchronization

Clock Synchronization: Basics

A clock in a computer is actually a timer – typically, oscillating quartz

with a counter and a holding register.
Each oscillation decrements the counter by one and when the counter
gets to zero, an interrupt is generated and the counter is reloaded from
the register.
Each interrupt is called a clock tick.
Synchronization in distributed systems is much more difficult than in
uniprocessor systems
Suppose we have the UNIX make program compiling the source file
output.c in another machine and generating the object file output.o.
In this example, make will not call the compiler.

7 / 38
 Clock Synchronization Clock Synchronization

Clock Synchronization: Basics

Problems with multiple computers (distributed systems):
Clock skew: The difference between two computers clocks.
Clock drift: Difference in the rate at which two clocks count the time.
- Due to physical differences in the used oscillation crystals, but also heat,
humidity, voltage etc.
- Accumulated drift can lead to significant skew.
Solution: Clock Synchronization
- UTC is broadcast to a UTC-receiver node in a distributed system through
radio signal or earth satellites and accessed through global positioning
systems (GPS).
- Satellites can give an accuracy of about 0.5ms, and even better.
Suppose we have a distributed system with a UTC-receiver somewhere in it, and we still
have to distribute its time to each machine.
Every machine has a timer that generates an interrupt H times per second.
There is a clock in machine p that ticks on each timer interrupt, and denote
the value of that clock by Cp (t), where t is UTC time.
Ideally, we have that for each machine p, Cp (t) = t, or, in other words,
dC
dt = 1.
8 / 38
 Clock Synchronization Clock Synchronization

Clock Synchronization: Basics

Goal: Never let two clocks in any system differ by more than δ time
units
Synchronize(in software) at least every δ /(2ρ) seconds , where ρ is the
maximum drift rate
In practice: 1 − ρ ≤ dC
dt ≤ 1 + ρ.

dC
>1
dt dC
Clock time, C =1
dt

ck

k
oc
clo

cl
dC

ct
st <1

rfe
ck dt
Fa

Pe
clo
w
Slo

UTC, t

9 / 38
 Clock Synchronization Clock Synchronization

Clock Synchronization: Algorithms

Cristians Algorithm
Periodically, each machine sends a message to the time server asking for
the current time.
The time server is passive.
The time server responds as fast as it can with a message containing its
current time, CUTC .
When the client gets the reply, it just sets its clock to CUTC .

10 / 38
 Clock Synchronization Clock Synchronization

Clock Synchronization: Algorithms

Cristians Algorithm
Major Problem: CUTC may be smaller than the clients current value of C.
Time must never run backward.
Solution: Introduce the change gradually
When slowing down, the interrupt routine adds less msec to the time until
the correction has been made.
When moving faster, add more msec at each interrupt.
Minor Problem: it takes a nonzero amount of time for the time servers
reply to get back to the client.
Solution: Estimate message propagation time T.
In the absence of any other information,T = (T1 − T0)/2.
If the time taken by time server to handle the interrupt and process the
incoming message (I) is known, T = (T1 − T0)/2.
To improve accuracy:
- The average of a series of measurement can be taken.
- Measurements in which T1-T0 exceeds a threshold are discarded for being
victims of network congestion.
- Messages that come back fastest can be taken to be themost accurate.
11 / 38
 Clock Synchronization Clock Synchronization

Clock Synchronization: Algorithms

The Berkeley Algorithm

The approach is the exact opposite of Cristians Algorithm.
A time server polls every machine periodically to ask what time it is
there.
The time server is active.
The server may not be able to access UTC.
Based on the answers, it computes an average time and tells machines to
adjust their time.
Instead of sending the updated current time, the server only sends the
amount to be adjusted.
Clocks that are running fast are slowed down.
Clocks running slow jump forward.

12 / 38
 Clock Synchronization Clock Synchronization

Clock Synchronization: Algorithms

The Berkeley Algorithm: Example

a The time server asks all the other machines for their clock values using a polling
mechanism (and providing the current time).
b The machines answer, indicating how they differ from the time sent.
c The time server tells everyone how to adjust their clock based on a calculation
of the average time value.
13 / 38
 Clock Synchronization Clock Synchronization

Clock Synchronization: Algorithms

Network Time Protocol

Both Cristians and the Berkeley Algorithm are centralized algorithms.
Decentralised algorithms also exist, and the Internets Network Time Protocol
(NTP) is the best known and most widely implemented.
NTP enables clients across the Internet to be synchronized accurately to UTC
It uses a hierarchy of UTC-synchronized servers.

Primary Server: has UTC clock

Secondary Server: connected to

primary

Etc.

14 / 38
 Clock Synchronization Clock Synchronization

Clock Synchronization: Algorithms

Network Time Protocol: Modes of Synchronization
Multicast:
one computer periodically multicasts time information to all other
computers on the network
these adjust their clock assuming a very small transmission delay
only suitable for high speed LANs, yielding low but usually acceptable
synchronization
Procedure-call:
similar to Christians protocol
server accepts requests from clients
applicable where higher accuracy is needed, or where multicast is not
supported by the networks hardware and software
Symmetric:
to be used where highest accuracy is needed
synchronization between peer servers
a NTP server exchanges with several peers in addition to with parent
15 / 38
 Logical Clock

Logical Clock

Happens-Before Relationship
Lamports Logical Clocks
Vector Clocks

16 / 38
 Logical Clock Happens-Before Relationship

Happens-Before Relationship

Sometimes, the only thing needed is that there is a shared time,

regardless of absolute time.
So, algorithms based on active servers polling other servers to find out
the average time, and the required estimated corrections as well.
No machine is required to have UTC time.
Synchronization is made based on relative time.
Such clocks are referred to as logical clocks.
Note that (with this mechanism) there is no requirement for relative time
to have any relation to the real time.
Whats important is that the processes in the distributed system agree on
the ordering in which certain events occur.
A Happens-Before Relationship between events introduces a partial
ordering of events in a system with concurrently operating processes.

17 / 38
 Logical Clock Happens-Before Relationship

Happens-Before Relationship

A happens-before relation ->is defined as follows:

If A and B are events in the same process, and A occurs before B, then we
can state that:
A ”happens-before” B is true
A ->B
If A is the event of a message being sent by one process, and B is the event
of the same message being received by another process, then
A ”happens-before” B is true
A ->B
If A->B and B->C, then
A ”happens-before” C is true
A ->C
When neither A->B nor B->A can be observed, then nothing can be
said on their ordering.
A and B are said to be concurrent.

18 / 38
 Logical Clock Happens-Before Relationship

Lamports Logical Clocks

Problem:
How do we maintain a global view on the systems behavior that is
consistent with the happened-before relation?
Solution:
Attach a timestamp C(e) to each event e, satisfying the following
properties:
- Property P1: If a and b are two events in the same process, and a ->b, then
we demand that C(a) <C(b).
- Property P2: If a corresponds to sending a message m, and b to the receipt of
that message, then also C(a) <C(b).
Problem:
How do we attach a timestamp to an event when theres no global clock?
Lamports solution:
The receiving process adjusts its clock forward to one more than the
sending timestamp value.
This allows the happens-before relation to hold, and also keeps all the
clocks running in a synchronized state.
The clocks are all kept in sync relative to each other.
19 / 38
 Logical Clock Happens-Before Relationship

Lamports Logical Clocks

Algorithm
Each process Pi maintains a local counter Ci and adjusts this counter according to the
following rules:
1 Before executing an event, Pi executes
Ci ← Ci + 1
2 When process Pi sends a message m to Pj , after having executed the
previous step, Pj sets ms timestamp ts(m) as
ts(m) ← Ci
3 Upon the receipt of a message m, process Pj adjusts its own local counter
as
Cj ← max{Cj , ts(m)}
It then executes the first step and delivers the message to the application.
Note
Property P1 is satisfied by Step (1).
Property P2 is satisfied by Steps (2) and (3).
Concurrent events can be avoided by breaking ties through process IDs.
20 / 38
 Logical Clock Happens-Before Relationship

amports Logical Clocks

P1 P2 P3 P1 P2 P3
0 0 0 0 0 0
6 m1 8 10 6 m1 8 10
12 16 20 12 16 20
18 24 m2 30 18 24 m2 30
24 32 40 24 32 40
30 40 50 30 P2 adjusts 40 50
36 48 60 36 its clock 48 60
42 56 m3 70 42 61 m3 70
48 64 80 48 69 80
54 m4 72 90 70 m4 77 90
60 80 100 76 P1 adjusts 85 100
its clock
(a) (b)

a Three processes, each with its own clock running at different rates
b Lamports algorithm corrects the clocks

21 / 38
 Logical Clock Happens-Before Relationship

Lamports Logical Clocks

Position of the Clocks

Adjustments of local clocks take place in the middleware layer
Application layer

Application sends message Message is delivered to application

Adjust local clock Adjust local clock Middleware layer

and timestamp message

Middleware sends message Message is received

Network layer

22 / 38
 Logical Clock Happens-Before Relationship

Lamports Logical Clocks: Totally Ordered Multicasting

Problem
We sometimes need to guarantee that concurrent updates on a replicated database are
seen in the same order everywhere.
Example:
P1 adds $100 to an account (initial value: $1000)
P2 increments account by 1%
There are two replicas
Update 1 Update 2

Replicated database
Update 1 is Update 2 is
performed before performed before
update 2 update 1

Result
In absence of proper synchronization:
replica #1 ← $1111, while replica #2 ← $1110.
23 / 38
 Logical Clock Happens-Before Relationship

Lamports Logical Clocks: Totally Ordered Multicasting

Solution
Process Pi sends a multicast message msgi to all processes in the group, including the
sender, together with the senders timestamp.
Any incoming message Pj is queued in queuej , ordered by its timestamp.
Upon receipt of a message, a multicast acknowledgement is sent to the group.
Due to the ”happens-before” relationship, the timestamp of the
acknowledgement is always greater than that of the original message.
Pj passes (delivers) a message msgi to its application if:
msgi is marked as acknowledged by all the other processes;
msgi is at the head of queuej ;
for each process Pk , there is a message msgk in queuej with a larger
timestamp.
Note:
We are assuming that communication is reliable and FIFO ordered.
All the local queues at each process eventually contain the same contents,
as the timestamps ensure that the ”happens-before” relation holds.
In this way, all messages are deliveredprocessed in the same order
everywhere, and updates can occur in a consistent manner. 24 / 38
 Logical Clock Vector Clocks

Vector Clocks

Reading Assignment

25 / 38
 Mutual Execution Overview

Overview

In single processor systems, critical sections are protected using

semaphores, monitors, and similar constructs.
In distributed systems, mutual exclusion is implemented via:
Permission-based approach
- A process that wishes to access a shared resource must first get permission
from one or more other processes.
- Algorithms:
A Centralized Algorithm
A Decentralized Algorithm
A Distributed Algorithm
Token-based approach
Processes share a special message known as a token.
Token holder has right to access shared resource.
Algorithms: ⇒A Token Ring Algorithm

26 / 38
 Mutual Execution A Centralized Algorithm

A Centralized Algorithm
Use a central coordinator to simulate how it is done in a one-processor
system.

a Process 1 asks the coordinator for permission to access a shared

resource. Permission is granted.
b Process 2 then asks permission to access the same resource. The
coordinator does not reply.
c When process 1 releases the resource, it tells the coordinator, which then
replies to 2.
27 / 38
 Mutual Execution A Decentralized Algorithm

A Decentralized Algorithm
Each resource has n coordinators.
A client acquires the lock by sending a request to each coordinator.
Access requires a majority vote from m(> n/2)coordinators.
If a resource is already locked, then it will be rejected (as opposed to just
blocking).
When a coordinator crashes, it will recover quickly, but will have
forgotten about permissions it had granted.

28 / 38
 Mutual Execution A Distributed Algorithm

A Distributed Algorithm
When a process wants a resource, it creates a message with the name of
the resource, its process number, and the current (logical) time.
It then reliably sends the message to all processes and waits for an OK
from everyone.
When a process receives a message:
If the receiver is not accessing the resource and is not currently trying to
access it, it sends back an OK message to the sender.
If the receiver already has access to the resource, it simply does not reply.
Instead, it queues the request.
If the receiver wants to access the resource as well but has not yet done so,
it compares the timestamp of the incoming message with the one
contained in the message that it has sent everyone.
- If the incoming message has a lower timestamp, the receiver sends back an
OK.
- If its own message has a lower timestamp, it queues it up.
When done using a resource, send an OK on its queue and delete them
all from the queue.
29 / 38
 Mutual Execution A Distributed Algorithm

A Distributed Algorithm

Example

30 / 38
 Mutual Execution A Token Ring Algorithm

A Token Ring Algorithm

Processes are organized in a logical ring, and a token is passed between
them. The one that holds the token is allowed to enter the critical region
(if it wants to).
When a process wants to enter the critical section, it waits till it gets the
token, enters, holds the token, exits, and passes the token on.

a an unordered group of processes on a network

b a logical ring constructed in software
31 / 38
 Mutual Execution A Token Ring Algorithm

A Token Ring Algorithm

Comparison
Algorithm # msgs per Delay before entry Problems
entry/exit (in msg times)
Centralized 3 2 Coordinator crash
Decentralized 2mk + m, k = 1,2,... 2mk Starvation , low efficiency .
Distributed 2 (n – 1) 2 (n – 1) Crash of any process
Token ring 1 to ∞ 0 to n – 1 Lost token, process crash

k: number of attempts
m: number of coordinators contacted
n: the number of process.

32 / 38
 Election Algorithm Overview

Overview
Many distributed systems require a process to act as coordinator (for
various reasons, e.g. to handle mutual exclusion).
The question is how to select his special process dynamically.
In many systems the coordinator is chosen manually (e.g. file servers).
The selection can also be performed automatically by an ”election
algorithm”.
- The Bully Algorithm
- A Ring Algorithm
- Superpeer Selection
The overriding goal of all election algorithms is to have all the processes
in a group agree on a coordinator.
For simplicity, we assume the following:
Processes each have a unique, positive identifier.
All processes know all other process identifiers.
The process with the highest valued identifier is duly elected coordinator.
When an election ”concludes”, a coordinator has been chosen and is
known to all processes.
33 / 38
 Election Algorithm The Bully Algorithm

The Bully Algorithm

When any process notices that the current coordinator is down, it starts
an election.
Each process has an associated priority (weight).
The process with the highest priority should always be elected as the
coordinator.
Issue: How do we find the heaviest process?
Any process can just start an election by sending an election message to
all other processes (assuming you dont know the weights of the others).
If a process Pheavy receives an election message from a lighter process
Plight , it sends a take-over message to Plight . Plight is out of the race.
If a process doesnt get a take-over message back, it wins, and sends a
victory message to all other processes.
If a process that was previously down comes back up, it holds an election

If it happens to be the heaviest process currently running, it will win the

election.
The biggest guy in town always wins, hence the name bully algorithm.
34 / 38
 Election Algorithm The Bully Algorithm

The Bully Algorithm: Example

1 1 1
2 5 2 5 2 5
n
ctio OK Election
Ele

n
ctio
Election OK
4 6 4 6 4 6

Ele

n
El

tio
ec

ec
tio

El
n
0 3 0 3 0 3
7 7 7
Previous coordinator
has crashed
(a) (b) (c)

1 1
2 5 2 5

OK
Coordinator
4 6 4 6

0 3 0 3
7 7

(d) (e)

(a) Process 4 holds an election; (b) Processes 5 and 6 respond, telling 4

to stop;
(c) Now 5 and 6 each hold an election; (d) Process 6 tells 5 to stop; (e)
Process 6 wins and tells everyone. 35 / 38
 Election Algorithm A Ring Algorithm

A Ring Algorithm

The processes are ordered in a logical ring, with each process knowing
the identifier of its successor (and the identifiers of all the other
processes in the ring).
Process with the highest priority should be elected as coordinator.
When a process notices that a coordinator is down, it creates an election
message (which contains its own number) and starts to circulate the
message around the ring.
If a successor is down, the message is passed on to the next successor.
If a message is passed on, the sender adds itself to the list.
When it gets back to the initiator, everyone had a chance to make its
presence known.
The one with the highest priority is elected as coordinator.
The initiator sends a coordinator message around the ring containing a list
of all living processes.
36 / 38
 Election Algorithm A Ring Algorithm

A Ring Algorithm: Example

It does no harm to have extra messages circulating; at worst it consumes
a little bandwidth, but this not considered wasteful.
Example:

Here in this example, both 2 and 5 initiate election and elect 6:

Election initiated by 5: [5,6,0,1,2,3,4]
Election initiated by 2: [2,3,4,5,6,0,1]
37 / 38
Election Algorithm A Ring Algorithm

Any Question

38 / 38