Wait Events in RAC

Session 362
Arup Nanda
Longtime Oracle DBA
arup.blogspot.com
What is This?
• RAC Performance Tuning
– I teach a course – Performance Tuning in RAC
• Wait events are useful for understanding the bottlenecks
• All single instance wait events are applicable to RAC
• RAC has some special cases
• This session talks about those RAC-specific wait events
• This is just a subset of the events; not a comprehensive
list

Wait Events in RAC 2

What’s a Wait?
• A process in Oracle can only be in three states
– Doing something Useful (consuming CPU) ….. U
– Idle, waiting for some work to be assigned ….. I
– Waiting for something, e.g. ….. W
• a block from disk
• a lock
• a latch (could be waiting on CPU)
• Response time = U + I + W
• We must accurately measure each component time
before we decide what and how to tune

Wait Events in RAC

3
Cluster Coordination
Buffer Cache Buffer Cache

SCN1 SCN2

DBWR DBWR Checkpoint!

LMS LMS

Checkpoint!
Node 1 Node 2

Database

DBWR must get a lock on the database block before writing to

the disk. This is called a Block Lock.
RAC for Beginners
4
Cache Fusion
Will get it via
interconnect
session1 session2

Wants to
Instance 1 Instance 2
Has
modify it modified
5 it

Wait Events in RAC 5

Checking for Buffers
How exactly is this “check”
performed? Block
• By checking for a lock on the
block Grant Convert
• The request comes to the Grant Queue Queue
Queue of the block
SID1 SID5
• Someone checks that no other
instance has any lock
SID2 SID6
• Instance 1 can read from the
disk SID3 SID7
• i.e. Instance 1 is granted the
lock

Wait Events in RAC 6

Master Instance
• Only one instance holds the grant and
convert queues of a specific block
• This instance is called Master Instance of Block
that block Grant Convert
Queue Queue
• Master instance varies for each block
SID1 SID5
• The memory structure that shows the
SID2 SID6
master instance of a buffer is called Global
SID3 SID7
Resource Directory (GRD)
• That is replicated across all instances
• The requesting instance must check the
GRD to find the master instance
• Then make a request to the master
instance for the lock
Wait Events in RAC 7
Scenario 1 Session Instance 1 Instance 2

DB

• Session connected to Instance 1 wants to select a block

from the table
• Activities by Instance 1
1. Check its own buffer cache to see if the block exists
1. If it is found, can it just use it?
2. If it not found, can it select from the disk?
2. If not, then check the other instances
• How will it know which copy of the block is the best
source?
Wait Events in RAC 8
Cache Fusion
Buffer Cache Buffer Cache

SMON SMON
LMS LMS
message
buffer
Node 1 Node 2
When node 2 wants a buffer, it sends a message to the
other instance. The message is sent to the LMS (Lock
Management Server) of the other instance. LMS then
sends the buffer to the other instance. LMS is also called
Global Cache Server (GCS).
Wait Events in RAC 9
Grant Scenario 2
1. Check its buffer cache to see if the block exists
2. And the buffer is found. Can Instance1 use it?
Not really. The buffer may be old; it may have been changed
3. LMS of node1 sends message to master of the buffer
3. Master checks the GES and doesn’t sees any lock
4. Instance 1 is granted the global block lock
5. No buffer actually gets transferred

Wait Events in RAC 10

Grant Scenario 3
• Instance 1 is the master
– Then it doesn’t have to make a request for the grant
• In summary, here are the possible scenarios when
Instance1 requests a buffer
– Instance1 is the master; so no more processing is
required
– No one has the lock on the buffer, the grant is made
by the master immediately
– Another instance has the buffer in an incompatible
mode. It has to be changed.

Wait Events in RAC 11

Buffer States and Locks
• Buffers can be gotten in two states
– Current – when the intention is to modify
• Shared Current – most recent copy. One copy per
instance. Same as disk
• Exclusive Current – only one copy in the entire
cluster. No shared current present
– CR – when the intention is to only select
• Locks facilitate the state enforcement
– XCUR for Exclusive Current
– SCUR for Shared Current
– No locking for CR
Wait Events in RAC 12
Placeholder Event
• When the buffer is first requested, the session does not
know which of the three paths it will go on to
• Therefore it is assigned a placeholder event
• This event is known as gc cr block request (for Consistent
Read requests)
• If the request is made for the buffer in Current mode, the
event is gc current block request
• When one of the three options is chosen, the appropriate
event replaces the placeholder event

Wait Events in RAC 13

Grant Event
• If the session merely requests a grant from the master:
– It waits with the gc cr|current grant 2-way event
– gc cr grant 2-way, for requests for buffers in Consistent
Read mode
– gc current grant 2-way, for requests for buffers in
Current mode

Wait Events in RAC 14

gc current|cr grant 2-way
Block
gc file
db current block read
grant
scattered 2-way
request Grant Conver
Queue t Queue

SID1 SID5

SID2 SID6

Session SID3 SID7

re
qu
te d

e st
ra n
g

LMS LMS
GRD GES
Instance 2

Instance 1
Database

Wait Events in RAC 15

Block Event
• After the request is made
– and assuming that the buffer is not in the local cache
– the buffer may be found in another instance
– Requestor requests the buffer from the holding
instance
• Possibilities:
– Holder is also the master of that buffer
– Holder is not the master
• If the holder is the master, the requesting session waits
with the event gc cr|current block 2-way

Wait Events in RAC 16

gc current|cr block 2-way
Block
gc
gccurrent
currentblock
block2-way
request Grant Conver
Queue t Queue

SID1 SID5

SID2 SID6

Session SID3 SID7

LMS LMS
GRD GES
Instance 2

Instance 1 Database

Wait Events in RAC 17

gc current|cr block 3-way
Block

gc current block 3-way Grant Conver

Queue t Queue

SID1 SID5

SID2 SID6

Session SID3 SID7

Master
Instance 2

Requestor Holder
Instance 1 Instance 3

Wait Events in RAC 18

Grant –vs- Get Events
Action Wait Event is
Block is requested by a session Placeholder event – gc cr|current block
request
Buffer lock is requested from the master gc cr|current grant 2-way

Buffer lock is granted, get from the disk db file scattered|sequential read

Buffer is requested from the holder which gc cr|current block 2-way

is the same as the master
Buffer is requested from the holder which gc cr|current block 3-way
is not the master

1. There is no 3-way grant event, since the request is made to the master
2. There is no 4-way block event; since there will a maximum of 2 hops:
requestor  master  holder

Wait Events in RAC 19

Interpretation
Scenario
The grant waits are very high
compared to block waits
Interpretation Tuning Implication
The requested blocks are in •Manually remaster the
the current instance; but the object?
master is another instance

The 2-way block waits are The cache fusion is taking •Faster interconnect
very high compared to 3-way place as expected, and most •Shorten the run queue
of the blocks are held and
mastered at the same
instance. But most of the
blocks are not found in the
local instance.
The 3-way block waits are Cache fusion is taking place; •Manually remaster the
high compared to 2-way but the master and the object
holder are usually different. •Application partitioning

Wait Events in RAC 20

gc current/cr block lost
• Lost blocks while being transferred to
the remote instance [in the
interconnect]
• Cause: may not be the network itself
• Cause #1 Network
– Network traffic drops the packets G Block
C
o

– Confirm from the ifconfig -a output

r n

gc
gccurrent
currentblock
block2-way
request
a v
n e
t r
Q t
u Q
e u
SID1 u SID5 e
e u
e

– If packets are dropping, this could

SID2 SID6

Session
Session SID3 SID7

be a cause
– Why?
LMS
LMS LMS
LMS
GRD
GRD GES
GES Instance
Instance 2
2

• Bad network configuration Instance

Instance 1
1 Database

• CPU used for network

processing
• Cause #2 CPU
– LMS process is CPU starved

Wait Events in RAC 21

gc current/cr block busy
• What it Means
– Session wants a block from the remote instance
– Remote instance delays preparing the block to
fulfill the request [CR or current] read
• Cause:
– Local delay on the remote instance
– Most likely: an I/O bottleneck on the remote
instance
• block is being accessed by some session which is CPU
starved
• LGWR has not written a buffer to redo yet
– Less likely: CPU starvation

Wait Events in RAC 22

gc current/cr block congested
• Meaning
– The instance has requested block from remote instance.
– Remote instance has prepared the block and shipped it
• but it has not reached the requesting instance within 1 ms.
• What could be cause – network bottleneck?
• Not necessarily. Causes:
– Long run queues, causing the LMS process to be
delayed in processing the incoming block
– Heavy paging due to memory deficiency. This causes
the blocks to be paged in before being processed.

Wait Events in RAC 23

Putting it All Together
• Every Oracle process is either
– Doing some productive work
– Waiting for some work to be given (idle)
– Waiting for some resource
• Understand the reason for the wait
– Devise a plan accordingly
• RAC related wait events are manifestations of these
issues mostly
• Caused by
– Network issues
– LMS being overloaded
– Blocks busy

Wait Events in RAC 24

Thank You! Session 362
• More Information:
– Blog: arup.blogspot.com
• 100 Things You Probably Didn't Know About Oracle
Database http://bit.ly/evr05e
– Twitter: arupnanda
Wait Events in RAC 25