DISTRIBUTED FILE SYSTEM [DFS]

ALLOWS ACCESS TO FILES & DATA ACROSS MULTIPLE PHYSICAL MACHINE

ENSURES DATA AVAILABILITY, REDUNDANCY & EFFICIENT DATA MANAGEMENT
KEY FEATURES
CONCURRENCY
REPLICATION
FAULT TOLERANCE
SCALABILITY
TRANSPARENCY
ISSUES
LACK OF GLOBAL KNOWLEDGE
CONSISTENCY
FAULT TOLERANCE
SCALABILITY
SECURITY
LATENCY
DATA INTEGRITY
PROCESS SYNCHRONISATION
HETEROGENEITY
MANAGEMENT
MONITORING
EXAMPLES
GOOGLE FILE SYSTEM [GFM]
HADOOP DISTRIBUTED FILE SYSTEM [HDFS]
CEPH

COHERENT DISTRIBUTED SHARED MEMORY [DSM]

SHARE COMMON MEMORY SPACE ACROSS MULTIPLE NODES AS IF ALL NODES HAVE ACCESS
TO SAME PHYSICAL MEMORY
MEMORY COHERENCE
ALL PROCESSORS SEE SAME VALUE FOR SHARED VALUE AT ALL TIMES
COHERENCE VS CONSISTENCY
COHERENCE PROTOCOLS
SNOOPING PROTOCOLS - RELY ON BROADCAST MEDIUM
WRITE INVALIDATE
WRITE UPDATE
DISTRIBUTED-BASED PROTOCOLS
CENTRALIZED DIRECTORY
DISTRIBUTED DIRECTORY
COHERENCE STATES [MESI]
MODIFIED
EXCLUSIVE
SHARED
INVALID
CHALLENGES
LATENCY
BANDWIDTH
SCALABILITY
FAULT TOLERANCE
RESOURCE MANAGEMENT
IMPLEMENTATION OF DSM
MEMORY MAPPING
COMMUNICATION MECHANISM
CONSISTENCY & COHERENCE PROTOCOLS
SYNCHRONIZATION
PERFORMANCE OPTIMIZATION
LOG-STRUCTURED FILE SYSTEM [LFS]
TREATS ITS STORAGE AS CONTINUOUS LOG
KEY FEATURES
WRITE OPTIMISATION
CRASH RECOVERY
GARBAGE COLLECTION
CHECKPOINTING
IMPLEMENTATION
SPRITE LFS
NILFS [NEW IMPLEMENTATION OF LFS]
WAFL [WRITE ANYWHERE FILE LAYOUT]

LFS VS TRADITIONAL FILE SYSTEM

ALGORITHM TO IMPLEMENT DSM

CENTRAL SERVER ALGORITHM

MAINTAINS ALL SHARED DATA
READ REQUEST & WRITE REQUEST
TIME-OUT & SEQUENCE NUMBER

MIGRATION ALGORITHM
DATA SHIPPED TO LOCATION OF DATA ACCESS REQUEST
ONLY ONE NODE ACCESS SHARED DATA AT A TIME
SUSCEPTIBLE TO THRASHING
INTEGRATE DSM WITH VIRTUAL MEMORY

READ REPLICATION ALGORITHM

REPLICATING DATA BLOCKS
ALLOW MULTIPLE NODE TO HAVE READ ACCESS OR ONE NODE TO READ & WRITE
ACCESS
ACCESS DATA CONCURRENTLY - IMPROVE SYSTEM PERFORMANCE
WRITE OPERATION - EXPENSIVE
KEEP TRACK OF LOCATION OF ALL COPIES

FULL REPLICATION ALGORITHM

ALLOW MULTIPLE NODES TO HAVE BOTH READ & WRITE ACCESS TO SHARED BLOCK
CONTROL ACCESS TO SHARED DATA TO MAINTAIN CONSISTENCY
GAP FREE SEQUENCES

DISTRIBUTED SCHEDULING [DS]

ALLOCATE TASK TO NODES TO ACHIEVE ~
OPTIMAL PERFORMANCE
LOAD BALANCING
EFFICIENT RESOURCE UTILISATION
MINIMUM EXECUTION TIME

KEY FEATURES
LOAD BALANCING
FAULT TOLERANCE
MINIMIZING LATENCY
MAXIMIZING THROUHPUT

DS ALGORITHM
 STATIC SCHEDULING ALGO
MAKE DECISION AT COMPILE TIME BEFORE EXECUTION
ALLOCATION OF TASK NOT CHANGE AT RUN TIME
EX~
ROUND ROBIN
TASK ASSIGNED IN CIRCULAR ORDER
RANDOM ASSIGNMENT
TASK ASSIGNED IN RANDOM ORDER
MIN MIN ALGO
TASK WITH MIN. COMPLETION TIME ON FASTEST AVAILABLE
NODE IS SCHEDULED FIRST

DYNAMIC SCHEDULING ALGO

MAKE DECISIONS AT RUN TIME
MORE FLEXIBILITY & ADAPTABILITY
EX~
LOAD BALANCING
TASK MIGRATION
AUCTION BASED SCHEDULING

IMPLEMENTING LOAD BALANCING WITH TASK MIGRATION

INITIALISATION
TASK ASSIGNMENT
LOAD MONITORING
TASK MIGRATION
CONSISTENCY & FAULT TOLERANCE

STRONGLY CONSISTENT SET OF CHECKPOINTS

CRUCIAL FOR ENSURING DATA INTEGRITY & SYSTEM RELIABILITY DURING RECOVERY
AFTER FAILURES

KEY POINTS
CHECKPOINTS
STRONG CONSISTENCY
GLOBAL SNAPSHOT

CHANDY LAMPORT ALGORITHM

ENSURES THAT GLOBAL SNAPSHOT OF SYSTEM IS CONSISTENT EVEN IN PRESENCE
OF ONGOING OPERATION
COMMUNICATION CHANNEL ARE FIFO
NODES COMMUNICATE BY PASSING MESSAGES

STEPS
INITIATE CHECKPOINTING
RECORD LOCAL STATE
RECORD CHANNEL STATE
COMPLETION

HANDLING FAILURE
STABLE STORAGE
CHECKPOINT COORDINATION
ROLLBACK MECHANISM
RECOVERY PROTOCOL