Advanced Database Design and Implementation

Instructor: Eric Umuhoza

Email: eric.umuhoza@gmail.com
Twitter: @EricUmuhoza

July 9, 2023

1 / 41
 Reliability Control

In this Lecture

• Persistence of memory and backup

• Buffer management
• Reliable transaction management
• Log management
• Recovery after malfunctions

”Things will go wrong in any given situation, if you give them a chance”

2 / 41
 Reliability Control

Persistence of Memories

• Main memory
– Not persistent
• Mass memory
– Persistent but can be damaged
• Stable memory
– Cannot be damaged (it is an abstraction)

⇒ Replication in several nonvolatile storage media

with independent failure modes

3 / 41
 Reliability Control

How to Guarantee Stable Memory

• On-line replication: mirroring of two disks

4 / 41
 Reliability Control

How to Guarantee Stable Memory

• Off-line replication: tape unit (backup)

5 / 41
 Reliability Control

Main Memory Management

Rationale
• Reuse of data in the buffer
• Deferred writing into the database

• The buffer is a large area of the main memory pre-allocated to

the DBMS and shared among the various transactions.
• In certain cases the entire dbms can be copied and managed in
the main memory.
6 / 41
 Reliability Control

Operating System Issues

• Operating system virtual memories allow the management of

page transfers between the secondary and main memory.
• But most OSs do not support the requirements of database
logging
– Cannot enforce log records prior to the output of database blocks
(see next slides)

DBMSs reimplement the buffer management

7 / 41
 Reliability Control

Buffer Management

• Based on four primitives:

– fix is used to load a page into the buffer. After the operation, the page
is allocate to a transaction.
– use primitive is used to gain access to the page previously loaded in
the buffer.
– unfix is used to de-allocate the page.
– force, synchronously transfers a page from the main memory to the
database.

8 / 41
 Reliability Control

Using the Buffer (Transactions)

• Follows the scheme
– fix
– repeat use until (end of transaction)
– unfix
• Pages are written by the buffer asynchronously
• flush
– This primitive is controlled by the buffer manager
– Asynchronously transfers a page from the main memory to the database

In practice, the primitives fix and use allow the loading into the
buffer and the reading of data, and the primitive force is used by the
transactions to write data in the secondary memory. Furthermore,
the flush primitive is used by the buffer manager itself to transfer to
the secondary memory the pages that are no longer valid and remain
inactive for a long time.

9 / 41
 Reliability Control

Buffer Management Policies

• Steal, used during the execution of the fix operation, allows the
buffer manager to select an active page allocated to another
transaction as a victim – page taken away from an active
transaction.
• No Steal, excludes this possibility.
• Force requires that all the active pages of a transaction are
transcribed in the secondary memory when at commit-work.
• No Force, entrusts the writing of the pages of a transaction to
the asynchronous mechanisms of the buffer manager.

Normally, the pair (No Steal, No Force) is preferred by the DBMSs,

as the No Steal policy is the easiest to carry out and the No Force
policy guarantees the higher efficiency.

10 / 41
 Reliability Control

Buffer Management Policies

There is also the possibility of anticipating the loading and unloading

times of the pages, by means of pre-fetching and pre-flushing policies:
• Pre-fetching
– anticipates reading of pages
– especially useful in sequential reading

• Pre-flushing
– anticipates writing of de-allocated pages
– useful for accelerating page fix

11 / 41
 Reliability Control

Architecture of the Reliability Manager

12 / 41
 Reliability Control

Reliability Control System

• The reliability control system ensures two fundamental properties

of transactions, atomicity and durability.
• The system is responsible for the writing of the log.
Log
Log is a permanent archive, which registers the various actions, car-
ried out by the DBMS.
Each write action on the database is protected by means of an ac-
tion on the log, so that it is possible to undo the actions following
malfunctions or failures preceding the commit, or redo these ac-
tions whenever their success is uncertain and the transactions have
performed a commit.

13 / 41
 Reliability Control

Reminder: Atomicity Requirements

• A transaction is an atomic transformation from the initial state

to the final state.
• Possible behaviors:
1 Commit work: SUCCESS
2 Rollback work or error before commit: UNDO
3 Fault after commit: REDO

14 / 41
 Reliability Control

Transaction: Unit of Recovery

• Recovery manager responsible for atomicity and durability

• If failure occurs between commit and database buffers being

flushed to secondary storage then, to ensure durability, recovery
manager has to redo, rollforward, transaction’s updates.

• If transaction had not committed at failure time, recovery

manager has to undo, rollback, any effects of that transaction
for atomicity.

15 / 41
 Reliability Control

Transactions and Recovery

• DBMS starts at time t0 , but fails at time tf . Assume data for

transactions T2 and T3 have been written to secondary storage
• T1 and T6 have to be undone
• In absence of any other information, recovery manager has to
redo T2 , T3 , T4 , and T5
16 / 41
 Reliability Control

Recovery Facilities

• DBMS should provide following facilities to assist with recovery:

– Backup mechanism, which makes periodic backup copies of database
– Logging facilities, which keep track of current state of transactions
and database changes
– Checkpoint facility, which enables updates to database in progress to
be made permanent
– Recovery manager, which allows DBMS to restore database to
consistent state following a failure

17 / 41
 Reliability Control

Transaction Log

• Sequential file consisting of records that describe the actions

carried out by the various transactions
• Written sequentially to the top block (top = current instant)

18 / 41
 Reliability Control

Main Function of the Log

• It records in the stable memory the actions carried out by the

various transactions under the form of state transitions
If UPDATE(U)
transforms O from value O1 to value O2
Then the log records:
BEFORE-STATE(U) = O1
AFTER-STATE(U) = O2

19 / 41
 Reliability Control

Using the Log

• After rollback-work or failure

– UNDO T1: O = O1

• After failure after commit

– REDO T1: O = O2

• Idempotency of UNDO and REDO

UNDO(T) = UNDO (UNDO(T))
REDO(T) = REDO (REDO(T))

20 / 41
 Reliability Control

Types of Log Records

• Records relevant to transactional commands:

– begin
– commit
– abort
• Records relevant to operations
– insert
– delete
– update
• Records relevant to recovery actions
– dump. A dump is a complete copy of the database (backup), which is
normally created when the system is not operative.
– checkpoint. A checkpoint is an operation that is carried out
periodically to record which transactions are active and to update
secondary memory relative to all completed transactions.

21 / 41
 Reliability Control

Types of Log Records

• Records relevant to transactional commands:

– B(T), C(T), A(T)
• Records relevant to operations
– I(T,O,AS), D(T,O,BS), U(T,O,BS,AS)
• Records relevant to recovery actions
– DUMP,CKPT(T1,T2,...,Tn)
• Record fields:
– T: transaction identifier
– O: object identifier
– BS, AS: before state, after state

22 / 41
 Reliability Control

Log and buffer management

• Log records are smaller than a page → the buffer manager tries
to write multiple log records at once since the cost of writing into
secondary memory is high
• But logs are lost if the system crashes!
• We must impose additional requirements on the recovery
techniques to ensure transaction atomicity

23 / 41
 Reliability Control

Transactional Rules

• Write–Ahead–Log
– Before-state parts of the log records must be written in the log
before carrying out the corresponding operation on the database
– Actions can be undone

• Commit Rule
– After-state parts of the log records must be written in the log before
carrying out the commit
– Actions can be redone

24 / 41
 Reliability Control

Writing onto Log and Database

• Writing onto the database before commit

– Requires writing (UNDO) in order to abort

25 / 41
 Reliability Control

Writing onto Log and Database

• Writing onto the database after commit

– Does not require writing in order to abort

26 / 41
 Reliability Control

Writing onto Log and Database

• Writing onto the database arbitrary

– Allows optimizing buffer management

27 / 41
 Reliability Control

In Case of Failure

• Soft failure
– Loss of the contents of the main memory
– Requires warm restart

• Hard failure
– Failure of secondary memory devices
– Requires cold restart

28 / 41
 Reliability Control

Fail-stop Failure Model

29 / 41
 Reliability Control

Checkpoint

• Consistent time point (in which all transactions write their data
from the buffer to the disk)
• All active transactions are recorded

30 / 41
 Reliability Control

Checkpoint

• Operation used to ”sum things up”, by simplifying the

subsequent restore operations

Aim: to record which transactions are active at a given moment

(and, dually, to confirm that the others either did not start or have
finished)

31 / 41
 Reliability Control

Checkpoint

• Several possibilities – the simplest is as follows:

– Acceptance of requests of any kind is suspended (writing, insertions,
..., commit, abort)
– All dirty pages connected to committed transactions are transferred to
mass storage (via force)
– The identifiers of the transactions in progress are recorded onto the log
synchronously (force)
– Acceptance of operations is resumed

• This way, we are sure that

1 For all committed transactions, the data are on mass storage
2 Transactions that are ”half-way” are listed in the checkpoint

32 / 41
 Reliability Control

Checkpoint

• With checkpoint at time tc , changes made by T2 and T3 have

been written to secondary storage:
– redo T4 and T5
– undo T1 and T6

33 / 41
 Reliability Control

Dump

• Time point in which a complete copy of the database is created

(typically during the night or the week-end)
• The presence of the dump is recorded

34 / 41
 Reliability Control

Warm Restart

• Log records are read starting from the checkpoint

• Transactions are divided into:
– UNDO set
– REDO set

• UNDO and REDO

35 / 41
 Reliability Control

Warm Restart

• UNDO
– Active transactions before commit
• REDO
– Active transactions after commit

36 / 41
 Reliability Control

Warm Restart

37 / 41
 Reliability Control

Example of Warm Restart

• B(T1)
• B(T2)
• U(T1,O1,B1,A1)
• I(T1,O2,A2)
• U(T2,O3,B3,A3)
• UNDO=(T1,T2,T3)
• B(T3)
REDO=()
• U(T3,O4,B4,A4)
• D(T3,O5,B5) • UNDO=(T1,T3,T4)
• CKPT(T1,T2,T3) REDO=(T2)
• C(T2)
• B(T4)
• U(T4,06,B6,A6)
• A(T4)
• Failure
38 / 41
 Reliability Control

Example of Warm Restart

• B(T1) UNDO=(T1,T2,T3)
• B(T2) REDO=(T2)
• U(T1,O1,B1,A1) O1 = B1
• I(T1,O2,A2) DELETE(O2)
• U(T2,O3,B3,A3) O3 = A3
• B(T3)
• U(T3,O4,B4,A4) O4 = B4
• D(T3,O5,B5) O5 = B
• CKPT(T1,T2,T3)
• C(T2)
• B(T4)
• U(T4,06,B6,A6) O6 = B6
• A(T4)
• Failure RESTART
39 / 41
 Reliability Control

Cold Restart

• Data are restored starting from the backup

• The operations recorded onto the log until the failure are
executed
• A warm restart is executed

40 / 41
Reliability Control

The End

41 / 41