SA Forum Extended Training Materials

Lock Service

SAI-AIS-LCK-B.02.01

Contents
This lesson addresses the following topics:
Defining, acquiring, and releasing cluster-wide locks by
processes on any node in the cluster Lock Service Model Lock resources Locks (PR and EX mode locks) Lock Service Types Lock Service Library Lifecycle Functions Lock Resource Functions Lock Functions Detecting and Handling, or Preventing, Deadlocks Orphan Locks and Lock Stripping Alarms and Notifications
Copyright 2006 Service Availability Forum, Inc

Lock Service
The Lock Service is a distributed lock service that allows different application processes on the same or different nodes in the cluster to compete for access to a shared resource in the cluster Examples of a shared resource (lock resource), include files,
databases, processes, components, hardware equipment

Access to a shared resource must be protected so that, e.g., one process does not try to read the resource while another process is writing it, or two processes do not try to write to the resource at the same time To protect access to a shared resource, a process must claim the lock on the resource and the Lock Service must grant the lock to the process before the process can access the resource
Copyright 2006 Service Availability Forum, Inc

Lock Service Example

The Lock Service Library linked into application processes and a Lock Service Server
This figure shows one particular implementation strategy.
Other implementations are possible; for example, the Lock Service Libraries might communicate directly with each other
Node U
Process P Component C saLckResourceLock(R) Lock Service Library

Node V
Process Q Component D

Node W
Lock Resource R

Node X

saLckResourceLock(R) Lock Service Library Lock Service Server

Copyright 2006 Service Availability Forum, Inc

Lock Service Model

A lock resource is a globally-named resource, access to which is controlled by a lock A lock is used to synchronize access to a shared resource by different application processes There are two lock modes:

Exclusive access: Exclusive lock (EX lock)

Only one process can hold the lock at the same time Shared access: Protected read lock (PR lock) One or more processes can hold the lock at the same time, and no process may hold an exclusive lock

Copyright 2006 Service Availability Forum, Inc

Lock Service Model

The Lock Service offers two optional features, which an implementation of the Lock Service may (but is not required to) support :

Deadlock detection
Deadlock Two processes hold two different locks, each of
which is required by the other process, so that neither process can make progress The Lock Service warns a process requesting a lock that the lock request will deadlock

Lock orphaning
Lock orphan A lock that was held by a process that has failed

and, thus, will not release the lock The Lock Service does not strip the locks implicitly; rather, the locks are purged explicitly by calling a Lock Service function

Copyright 2006 Service Availability Forum, Inc

Lock Service
Example of a resource being claimed by two processes
Process P on node U
saLckResourceLock() lock lockId1 granted saLckDispatch() Process P now holds the lock

Lock Service server

Process Q on node V

saLckResourceLockAsync()

saLckLockWaiterCallback()
saLckDispatch()

Process P releases the lock

saLckResourceUnlock() saLckLockGrant()

Process Q now holds the lock

Copyright 2006 Service Availability Forum, Inc

Lock Service Types

SaLckLockFlagsT type
SA_LCK_LOCK_NO_QUEUE
Requests that the Lock Service not queue the lock request if the lock request cannot be granted immediately. The Lock Service returns the status SA_LCK_LOCK_NOT_QUEUED in the status block Requests that the Lock Service not purge this lock if the process or node hosting the process holding the lock fails, or if the process holding the lock calls saLckResourceClose() against the locked resource or calls saLckFinalize() without first releasing the lock

SA_LCK_LOCK_ORPHAN

SaLckLockModeT enum
typedef enum{ SA_LCK_PR_LOCK_MODE, SA_LCK_EX_LOCK_MODE } SaLckLockModeT
Allows the requesting process to read from a resource while other processes simultaneously read from the same resource. No process can write to the resource while one or more PR locks are hold on the resource Allows the requesting process to read from, or write to, a resource while it prevents any other process from accessing that resource
Copyright 2006 Service Availability Forum, Inc

Lock Service Types

SaLckLockStatusT enum
typedef enum{ SA_LCK_LOCK_GRANTED, SA_LCK_LOCK_DEADLOCK, SA_LCK_LOCK_NOT_QUEUED, SA_LCK_LOCK_ORPHANED, SA_LCK_LOCK_NO_MORE, SA_LCK_LOCK_DUPLICATE_EX } SaLckLockStatusT GRANTED Lock request was granted in the mode requested DEADLOCK Lock request would cause deadlock. This value is returned only if the implementation supports the optional deadlock detection feature NOT_QUEUED Lock request is blocked and would have to be queued but the request was submitted with the SA_LCK_LOCK_NO_QUEUE flag ORPHANED Lock request could not be granted because the lock is an orphan. A now-failed lock holder specified the SA_LCK_LOCK_ORPHAN flag. This value is returned only if the implementation supports the optional orphan locks feature NO_MORE The Lock Service cannot support any more locks DUPLICATE_EX The process requesting an EX lock already holds a granted or pending EX or PR lock against the same lock resource, or the process requesting a PR lock already holds a granted or pending EX lock against the same lock resource
Copyright 2006 Service Availability Forum, Inc

Lock Service Library Lifecycle Functions Initializing and finalizing the Lock Service
SaAisErrorT saLckInitialize(

/* OUT*/ SaLckHandleT *lckHandle,

/* IN */ const SaLckCallbacksT *lckCallbacks, /* IN/OUT */ SaVersionT *version);

Initializes the Lock Service for the invoking process and registers the callbacks saLckLockGrantCallback() and saLckLockWaiterCallback() lckHandle is used for subsequent calls

SaAisErrorT saLckFinalize(
/* IN */ SaLckHandleT lckHandle);

Closes the association, represented by the lckHandle parameter, between the invoking process and the Lock Service
Releases resources and cancels pending callbacks

Copyright 2006 Service Availability Forum, Inc

10

Lock Service Library Lifecycle Functions

Getting the selection object and dispatching a callback
SaAisErrorT saLckSelectionObjectGet(

/* IN */

SaLckHandleT lckHandle,

Obtains the operating system handle, selectionObject, associated with the handle lckHandle This selectionObject is used to detect pending callbacks Invokes, in the context of the calling thread, pending callbacks for the handle lckHandle, as specified by the dispatchFlags parameter

/* OUT */ SaSelectionObjectT *selectionObject); SaAisErrorT saLckDispatch(

/* IN */
/* IN */

SaLckHandleT lckHandle,
SaDispatchFlagsT dispatchFlags);

Copyright 2006 Service Availability Forum, Inc

11

Lock Library Lifecycle Functions

Checking for optional Lock Service features
SaAisErrorT saLckOptionCheck( /* IN */ SaLckHandleT *lckHandle, /* OUT */ SaLckOptionsT *lckOptions); Determines if any of the optional Lock Service features is provided by this implementation of the Lock Service

lckHandle The handle, obtained from the saLckInitialize() function, that designates this particular initialization of the Lock Service
lckOptions Bitmap of the flags:

SA_LCK_OPT_ ORPHAN_LOCKS SA_LCK_OPT_DEADLOCK_DETECTION

If a desired optional feature is not supported by the implementation of the Lock Service, it is up to the application to determine what action to take

Copyright 2006 Service Availability Forum, Inc

12

Lock Resource Functions

Opening a lock resource synchronously
SaAisErrorT saLckResourceOpen( /* IN */ SaLckHandleT lckHandle, /* IN */ const SaNameT *lockResourceName, /* IN */ SaLckResourceOpenFlagsT resourceFlags, /* IN */ SaTimeT timeout, /* OUT */ SaLckResourceHandleT *lockResourceHandle); Opens a (cluster-wide) lock resource associated with lockResourceName for locking operations synchronously lockResourceName The name of the lock resource being requested that identifies a lock resource globally in the cluster resourceFlags Currently, only one flag SA_LCK_RESOURCE_CREATE is defined timeout The maximum time that the caller is willing to wait for this function to return lockResourceHandle The lock resource handle, assigned by the Lock Service and returned to the caller. The caller must use this handle in subsequent requests to lock, unlock, purge, and close this lock resource

Copyright 2006 Service Availability Forum, Inc

13

Lock Resource Functions

Opening

a lock resource asynchronously

Opens a (cluster-wide) lock resource associated with lockResourceName for locking operations asynchronously invocation Allows the invoking process to match this invocation of saLckResourceOpenAsync() with the corrresponding callback lockResourceName The name of the lock
resource being requested that identifies a lock resource globally in the cluster resourceFlags Currently, only one flag SA_LCK_RESOURCE_CREATE is defined

SaAisErrorT saLckResourceOpenAsync( /* IN */ SaLckHandleT lckHandle, /* IN */ SaInvocationT invocation, /* IN */ const SaNameT *lockResourceName, /* IN */ SaLckResourceOpenFlagsT resourceFlags);

Completion of the saLckResourceOpenAsync() function is signaled by an invocation of the associated callback

Copyright 2006 Service Availability Forum, Inc

14

Lock Resource Functions

Opening

a lock resource asynchronously

Informs the process that the operation requested by the invocation of saLckResourceOpenAsync() completes

typedef void (*SaLckResourceOpenCallbackT)( /* IN */ /* IN */ /* IN */ SaInvocationT invocation, SaLckResourceHandleT *lockResourceHandle, SaAisErrorT error);

invocation Allows the caller to match this invocation of saLckResourceOpenCallback() with the corresponding invocation of saLckResourceOpenAsync()
lockResourceHandle The handle to the lock resource, which is assigned by the Lock Service and returned to the caller, if the operation completes successfully error Indicates whether the saLckResourceOpenAsync() function was successful

Copyright 2006 Service Availability Forum, Inc

15

Lock Resource Functions

Closing

a lock resource
Deletes the association between the lockResourceHandle and the corresponding lock resource Any held or pending lock requests are canceled. Once all references to the lock resource have been closed, the Lock Service considers that the lock resource no longer exists However, if the implementation supports orphan locks, and there are orphan locks against the lock resource, the lock resource continues to exist until all of the orphan locks have been purged

SaAisErrorT saLckResourceClose( /* IN */ SaLckResourceHandleT *lockResourceHandle);

Copyright 2006 Service Availability Forum, Inc

16

Lock Functions
Requesting

a lock on a resource synchronously

Requests a lock on the lock resource, designated by lockResourceHandle, synchronously lockResourceHandle The handle that designates the lock resource to be locked lockId The identifier of the lock, returned by the Lock Service lockMode The requested lock mode lockFlags The requested lock flags waiterSignal Signal delivered to the holder of the lock blocking this request timeout The maximum amount of time within which the lock must be granted lockStatus The status of the lock, returned by the Lock Service After the lock is granted, it is possible to receive a lock waiter notification via saLckLockWaiterCallback() if the process specified this callback when it initialized the Lock Service via the saLckInitialize() call
17

SaAisErrorT saLckResourceLock( /* IN */ SaLckResourceHandleT lockResourceHandle, /* OUT */ SaLckLockIdT *lockId, /* IN */ SaLckLockModeT lockMode, /* IN */ SaLckLockFlagsT lockFlags, /* IN */ SaLckWaiterSignalT waiterSignal, /* IN */ SaTimeoutT timeout, /* OUT */ SaLckLockStatusT *lockStatus);

Copyright 2006 Service Availability Forum, Inc

Lock Functions
Requesting

a lock on a resource asynchronously

Requests a lock on the lock resource, identified by lockId, asynchronously lockResourceHandle The handle that designates the lock resource to be locked invocation Associates this lock request with the corresponding invocation of saLckLockGrantCallback() by the Lock Service lockId The identifier of the lock, returned by the Lock Service lockMode The requested lock mode lockFlags The requested lock flags waiterSignal Signal delivered to the holder of the lock blocking this request The return value indicates whether the Lock Service has accepted the request for processing, rather than the status of the lock The result of the operation is returned by the saLckLockGrantCallback() function
18

SaAisErrorT saLckResourceLockAsync(
/* IN */ SaLckResourceHandleT lockResourceHandle, /* IN */ SaInvocationT invocation, /* OUT */ SaLckLockIdT *lockId,

/* IN */
/* IN */ /* IN */

SaLckLockModeT lockMode,
SaLckLockFlagsT lockFlags, SaLckWaiterSignalT waiterSignal);

Copyright 2006 Service Availability Forum, Inc

Lock Functions
Granting

a lock on a resource
Callback invoked when the operation requested by the invocation of saLckResourceLockAsync() completes invocation Used to match this invocation of saLckLockGrantCallback() with the corresponding invocation of saLckResourceLockAsync() lockStatus The status of the lock error Error value supplied by the Lock Service. The lock is granted, and the lock status is defined, only if the error value is SA_AIS_OK Implementations that support deadlock detection will detect deadlocks and indicate in the lockStatus field that granting the lock would cause a deadlock
Copyright 2006 Service Availability Forum, Inc

typedef void (*SaLckLockGrantCallbackT)( /* IN */ /* IN */ /* IN */ SaInvocationT invocation, SaLckLockStatusT lockStatus, SaAisErrorT error);

19

Lock Functions
Lock

waiter callback
Callback invoked when a process holds the lock, identified by lockId, that is blocking another lock request waiterSignal Specified by a lock requestor for delivery to the lock holder blocking the request lockId The lock identifier that was returned by the Lock Service to the process through saLckResourceLock() or saLckResourceLockAsync() modeHeld The lock mode (i.e., PR or EX) in which the process holds the lock

typedef void (*SaLckLockWaiterCallbackT)( /* IN */ /* IN */ /* IN */ SaLckWaiterSignalT waiterSignal, SaLckLockIdT lockId, SaLckLockModeT modeHeld,

/* IN */

SaLckLockModeT modeRequested);

modeRequested The lock mode (i.e., PR or EX) requested, for which the lock request is blocked

Copyright 2006 Service Availability Forum, Inc

20

Lock Functions
Releasing

a lock
Releases a lock on the lock resource, identified by lockId, synchronously lockId The identifier of the lock to be released. If the lockId identifies a pending lock request, the pending lock request will be canceled timeout The time by which the saLckResourceUnlock() function must complete

SaAisErrorT saLckResourceUnlock( /* IN */ SaLckLockIdT lockId, /* IN */ SaTimeoutT timeout);

SaAisErrorT saLckResourceUnlockAsync( Releases a lock on the lock resource, identified by lockId, asynchronously /* IN */ SaInvocationT invocation, invocation Allows the caller to match this /* IN */ SaLckLockIdT lockId); invocation of saLckResourceUnlock() with the corresponding callback lockId The identifier of the lock to be released The result of the operation is returned in the saLckResourceUnlockCallback() function, supplied in saLckInitialize()
Copyright 2006 Service Availability Forum, Inc

21

Lock Functions
Issuing

a callback and purging a lock

Callback invoked when the operation requested by the invocation of saLckResourceUnlockAsync() completes invocation Used to match this invocation of saLckResourceUnlockCallback() with the corresponding invocation of saLckResourceUnlock() and to determine the resource and lock for this unlock request error An error value supplied by the Lock Service. SA_AIS_OK indicates that the unlock request completed successfully Purges all orphaned locks held on a lock resource lockResourceHandle The handle to the lock resource on which one or more orphaned locks are held Orphaned locks are locks that were acquired with the SA_LCK_LOCK_ORPHAN flag set and that have not been unlocked properly
22

typedef void (*SaLckResourceUnlockCallbackT)( /* IN */ SaInvocationT invocation, /* IN */ SaAisErrorT error);

SaAisErrorT saLckLockPurge( /* IN */ SaLckResourceHandleT lockResourceHandle);

Copyright 2006 Service Availability Forum, Inc

Lock Service
Example of a resource being claimed by two processes
Process P on node U
saLckResourceLock() lock lockId1 granted saLckDispatch() Process P now holds the lock

Lock Service server

Process Q on node V

saLckResourceLockAsync()

saLckLockWaiterCallback()
saLckDispatch()

Process P releases the lock

saLckResourceUnlock() saLckLockGrant()

Process Q now holds the lock

Copyright 2006 Service Availability Forum, Inc

23

Deadlock
To prevent deadlock, processes that use the Lock Service can choose to define a priority order 0,1,...,n for the locks. The processes claim locks in priority order If a lock with priority j is held and another lock with priority k
is claimed, then k must be greater than j

Optionally, the Lock Service detects deadlocks, and returns to a lock requestor SA_LCK_LOCK_DEADLOCK as the lockStatus to indicate that claiming a lock will cause deadlock. The Lock Service does not resolve deadlocks It is the responsibility of processes that uses the Lock Service to ensure that invocations of lock requests do not result in deadlock
Copyright 2006 Service Availability Forum, Inc

24

Deadlock
Example of deadlock
Process P
lock lockId1 granted

Lock Service

Process Q

saLckResourceLock(..,lockId1,..)

saLckResourceLock(..,lockId2,..) lock lockId2 granted

saLckResourceLock(..,lockId2,..)

Waiting for lock lockId2.

saLckResourceLock(..,lockId1,..) SA_LCK_LOCK_DEADLOCK

Deadlock !!!
Copyright 2006 Service Availability Forum, Inc

25

Lock Stripping
A lock can be stripped from its holder under the following conditions:
The process owning the lock fails The node that hosts the process owning the lock fails The process owning the lock calls the saLckResourceClose()
function of the Lock Service for the particular locked resource, or it calls the saLckFinalize() function of the Lock Service without first releasing all of its locks

Copyright 2006 Service Availability Forum, Inc

26

Lock Stripping
Lock stripping without orphan locks
Process P Lock Service
Lock lockId1 is stripped from process P
saLckResourceLock(..,lockId1,..) Lock lockId1 granted

Process Q

saLckResourceLock(..,lockId1,..) Lock lockId1 granted

Process crashes

If the lock is a PR lock and other processes are already holding the lock, no change is reported to the other lock holders

Copyright 2006 Service Availability Forum, Inc

27

Lock Stripping
Lock stripping with orphan locks
SA_LCK_LOCK_ORPHAN

flag set in the lock call

Process P

Lock Service

Process Q

saLckResourceLock(..,lockId1,..) Lock lockId1 granted

Process crashes

saLckResourceLock(..,lockId1,..) Lock lockId1 is not granted SA_LCK_LOCK_ORPHAN

is returned in lock status

If the implementation supports orphan locks and the lock requestor specified the SA_LCK_LOCK_ORPHAN flag, the lock is not stripped; rather, it remains in the grant queue where it can block other lock requests The Lock Service provides an API to purge orphan locks held on a lock resource
Copyright 2006 Service Availability Forum, Inc

28

Alarms and Notifications

The Lock Service produces certain alarms and notifications to convey important information about its operational and functional state to an administrator or a management system The Lock Service alarms and notifications include: Lock Service Impaired Deadlock Detected the notification class identifier SaNtfClassIdT In The majorId field is set to SA_SVC_LCK The minorId field is set for each individual notification with
the following range bounds:

Alarms: 0x01 0x64 State change notifications: 0x65 0xC8 Object change notifications: 0xC9 0X12C Attribute change notifications: 0x12D 0x190

Copyright 2006 Service Availability Forum, Inc

29

Lock Service Alarms

Lock Service Impaired
NTF Attribute Name Event Type
Notification Object Probable Cause

Attribute Type (X.73Y SA Forum Recommendation or NTF) Recommended Value Mandatory

Mandatory Mandatory

SA_NTF_ALARM_ COMMUNICATION
LCK service, same as notifying object Applicable value from SaNtfProbableCauseT enum

Perceived Severity

Mandatory

Applicable value from SaNtfSeverityT enum

Copyright 2006 Service Availability Forum, Inc

30

Lock Service Alarms

Deadlock Detected
NTF Attribute Name
Event Type Notification Object

Attribute Type (X.73Y Recommendation or NTF)

Mandatory Mandatory

SA Forum Recommended Value

SA_NTF_ALARM_ PROCESSING LDAP DN of the lock resource on which the deadlock was detected Applicable value from SaNtfProbableCauseT enum in the Notification Service Application value from SaNtfSeverityT enum in the Notification Service
31

Probable Cause

Mandatory

Perceived Severity

Mandatory

Copyright 2006 Service Availability Forum, Inc

Summary
In this lesson you have learned about:
Defining, acquiring, and releasing cluster-wide locks Lock Service Model
Lock resources Locks (PR and EX mode locks) Lock Service Types

Lock Service Library Lifecycle Functions

Lock Resource Functions Lock Functions

Detecting and handling, or preventing, deadlocks

Orphan locks and lock stripping Alarms and notifications
Copyright 2006 Service Availability Forum, Inc

32