Rac Tuning Sig 2004

Oracle RAC SIG 2004
Performance Monitoring & Tuning for RAC

Rich Niemiec, TUSC
Special Thanks: Janet Bacon, Mike Ault, Madhu Tumma, Janet Dahmen, Randy Swanson, Rick Stark , Sohan DeMel Erik Peterson and Kirk McGowan
A TUSC Presentation 1
Audience Knowledge
Goals
Overview of RAC & RAC Tuning Target RAC tips that are most useful
Non-Goals
Learn ALL aspects of RAC
A TUSC Presentation
Overview
RAC Overview Tuning the RAC Cluster Interconnect Monitoring the RAC Workload Monitoring RAC specific contention Whats new in 10g Summary
A TUSC Presentation
Overview of Oracle9i RAC
Many instances of Oracle running on many nodes All instances share a single physical database and have common data & control files Each instance has its own log files and rollback segments (UNDO Tablespace) All instances can simultaneously execute transactions against the single database Caches are synchronized using Oracles Global Cache Management technology (Cache Fusion)
Oracle9i Database Clusters
Start small, grow incrementally Scalable AND highly available NO downtime to add servers and disk
Real Application Clusters
Server 1 Instance A
Server 2 Instance A
Database A
SERVER failure - your database failures available Protect from SERVER remains
E-Business Suite Scalability with Oracle9i RAC

Oracle11i E-Business Suite Benchmark
7,000 6,000 5,000
84% Scalability
6,496 5,433 4,368* 2,296*
# Users
4,000 3,000 2,000 1,000 0

1 Node
1,288
2 Nodes 4 Nodes 5 Nodes 6 Nodes
*Audited
Running on HP Computers
Analysis of Performance Issues

Normal database Tuning and Monitoring RAC Cluster Interconnect Performance Monitoring Workload Monitoring RAC specific contention
A TUSC Presentation
Normal Database Tuning and Monitoring
Prior to tuning RAC specific operations, each instance should be tuned separately.
APPLICATION Tuning DATABASE Tuning OS Tuning
You can begin tuning RAC
A TUSC Presentation
Tuning the RAC Cluster Interconnect
Your primary tuning efforts after tuning each instance individually should focus on the processes that communicate through the cluster interconnect.
Global Services Directory Processes
GES Global Enqueue Services GCS Global Cache Services
A TUSC Presentation
10
Global Cache Services (GCS) Waits

Indicates how efficiently data is being transferred over the cluster interconnect. The critical RAC related waits are: global cache busy
A wait event that occurs whenever a session has to wait for an ongoing operation on the resource to complete.
buffer busy global cache A wait event that is signaled when a process has
to wait for a block to become available because another process is obtaining a resource for this block.
buffer busy global CR
Waits on a consistent read via the global cache.

How:
1) Query V$SESSION_WAIT to determine whether or not any sessions are experiencing RAC related waits. 2) Identify the objects that are causing contention for these sessions. 3) Modify the object to reduce contention.
1)
Query v$session_wait to determine whether or not any sessions are experiencing RAC related waits.
SELECT inst_id, event, p1 FILE_NUMBER, p2 BLOCK_NUMBER, WAIT_TIME FROM v$session_wait WHERE event in ('buffer busy global cr', 'global cache busy', 'buffer busy global cache');
A TUSC Presentation
13
The output from this query should look something like this:
The block number and file number will indicate the object the requesting instance is waiting for.
INST_ID ------1 2
EVENT FILE_NUMBER BLOCK_NUMBER WAIT_TIME ----------------------------------- ----------- ------------ ---------global cache busy 9 150 15 global cache busy 9 150 10
A TUSC Presentation
14
2)
Identify objects that are causing contention for these sessions by identifying the object that corresponds to the file and block for each file_number/block_number combination returned:
SELECT owner, segment_name, segment_type FROM dba_extents WHERE file_id = 9 AND 150 between block_id AND block_id+blocks-1;
A TUSC Presentation
15
The output will be similar to:

OWNER SEGMENT_NAME SEGMENT_TYPE ---------- ---------------------------- --------------SYSTEM MOD_TEST_IND INDEX
A TUSC Presentation
16
3) Modify the object to reduce the chances for application contention.
Reduce the number of rows per block Adjust the block size to a smaller block size Modify INITRANS and FREELISTS
A TUSC Presentation
17
Tuning the RAC Cluster Interconnect CR Block Transfer Time
Block contention can be measured by using block transfer time, calculated by:
Accumulated round-trip time for all requests for consistent read blocks.
global cache cr block receive time global cache cr blocks received

Total number of consistent read blocks successfully received from another instance.
Use a self-join query on GV$SYSSTAT to compute this ratio:

COLUMN COLUMN PROMPT SELECT "AVG RECEIVE TIME (ms)" FORMAT 9999999.9 inst_id format 9999 GCS CR BLOCKS b1.inst_id, b2.value "RECEIVED", b1.value "RECEIVE TIME", ((b1.value / b2.value) * 10) "AVG RECEIVE TIME (ms)" gv$sysstat b1, gv$sysstat b2 b1.name = 'global cache cr block receive time' b2.name = 'global cache cr blocks received' b1.inst_id = b2.inst_id;
FROM WHERE AND AND
INST_ID RECEIVED RECEIVE TIME AVG RECEIVE TIME (ms) ------- ---------- ------------ --------------------1 2791 3287 11.8 2 3760 7482 19.9
Problem Indicators:
High Transfer Time One node showing excessive transfer time
Use OS commands to verify cluster interconnects are functioning correctly.
A TUSC Presentation
20
Tuning the RAC Cluster Interconnect CR Block Service Time
Measure the latency of service times. Service time is comprised of consistent read build time, log flush time, and send time.
global cache cr blocks served global cache cr block build time global cache cr block flush time global cache cr block send time
Number of requests for a CR block served by LMS The time that the LMS process requires to create a CR block on the holding instance. Time waited for a log flush when a CR request is served (part of the serve time) Time required by LMS to initiate a send of the CR block.
Query GV$SYSSTAT to determine average service times by instance:

SELECT a.inst_id "Instance", (a.value+b.value+c.value)/decode(d.value,0,1, d.value) "LMS Service Time" FROM gv$sysstat A, gv$sysstat B, gv$sysstat C, gv$sysstat D WHERE A.name = 'global cache cr block build time' AND B.name = 'global cache cr block flush time' AND C.name = 'global cache cr block send time' AND D.name = 'global cache cr blocks served' AND B.inst_id = A.inst_id AND C.inst_id = A.inst_id AND D.inst_id = A.inst_id ORDER BY a.inst_id; A TUSC Presentation 22
Result:
Instance LMS Service Time --------- ---------------1 1.07933923 2 .636687318
Instance 1 is on a faster node and serves blocks faster.
The service time TO Instance 2 is shorter!

Instance 2 is on the slower node.
A TUSC Presentation
23

Query GV$SYSSTAT to drill-down into service time for individual components:
SELECT A.inst_id "Instance", (A.value/D.value) "Consistent Read Build", (B.value/D.value) "Log Flush Wait",(C.value/D.value) "Send Time FROM GV$SYSSTAT A, GV$SYSSTAT B, GV$SYSSTAT C, GV$SYSSTAT D WHERE A.name = 'global cache cr block build time' AND B.name = 'global cache cr block flush time' AND C.name = 'global cache cr block send time' AND D.name = 'global cache cr blocks served' AND B.inst_id=a.inst_id AND C.inst_id=a.inst_id AND D.inst_id=a.inst_id ORDER BY A.inst_id;
Instance Consistent Read Build Log Flush Wait Send Time --------- --------------------- -------------- ---------1 .00737234 1.05059755 .02203942 2 .04645529 .51214820 .07844674
Tuning the RAC Cluster Interconnect OS Troubleshooting Commands
Monitor cluster interconnects using OS commands to find:

Large number of processes in the run state waiting for cpu or scheduling Platform specific OS parameter settings that affect IPC buffering or process scheduling Slow, busy, faulty interconnects. Look for dropped packets, retransmits, CRC errors. Ensure you have a private network Ensure inter-instance traffic is not routed through a public network
Tuning the RAC Cluster Interconnect OS Troubleshooting Commands
Useful OS commands in a Sun Solaris environment to aid in your research are:

$ $ $ $ $ netstat l netstat s sar c sar q vmstat
A TUSC Presentation
26
Tuning the RAC Cluster Interconnect Global Performance Views for RAC
Global Dynamic Performance view names for Real Application Clusters are prefixed with GV$.
GV$SYSSTAT, GV$DML_MISC, GV$SYSTEM_EVENT, GV$SESSION_WAIT, GV$SYSTEM_EVENT contain numerous statistics that are of interest to the DBA when tuning RAC. The CLASS column of GV$SYSSTAT tells you the type of statistic. RAC related statistics are in classes 8, 32 and 40.
Tuning the RAC Cluster Interconnect Your Analysis Toolkit
RACDIAG.SQL is an Oracle-provided script that can also help with troubleshooting.

RACDIAG.SQL can be downloaded from Metalink or the TUSC website, www.tusc.com. STATSPACK combined with the queries illustrated in this presentation will provide you with the tools you need to effectively address RAC system performance issues.
A TUSC Presentation
28
Tuning the RAC Cluster Interconnect Undesirable Statistics
As mentioned, the view GV$SYSSTAT will contain statistics that indicate the performance of your RAC system. The following statistics should always be as near to zero as possible:
global cache blocks lost global cache blocks corrupt
Block losses during transfers. May indicate network problems. Blocks that were corrupted during transfer. High values indicate an IPC, network, or hardware problem.
A TUSC Presentation
29
SELECT A.VALUE "GC BLOCKS LOST 1", B.VALUE "GC BLOCKS CORRUPT 1", C.VALUE "GC BLOCKS LOST 2", D.VALUE "GC BLOCKS CORRUPT 2" FROM GV$SYSSTAT A, GV$SYSSTAT B, GV$SYSSTAT C, GV$SYSSTAT D WHERE A.INST_ID=1 AND A.NAME='global cache blocks lost' AND B.INST_ID=1 AND B.NAME='global cache blocks corrupt' AND C.INST_ID=2 AND C.NAME='global cache blocks lost' AND D.INST_ID=2 AND D.NAME='global cache blocks corrupt' GC BLOCKS LOST 1 GC BLOCKS CORRUPT 1 GC BLOCKS LOST 2 GC BLOCKS CORRUPT 2 ---------------- ------------------- ---------------- ------------------0 0 652 0
Instance 2 is experiencing some lost blocks.

Take a closer look to see what is causing the lost blocks:

SELECT A.INST_ID "INSTANCE", A.VALUE "GC BLOCKS LOST", B.VALUE "GC CUR BLOCKS SERVED", C.VALUE "GC CR BLOCKS SERVED", A.VALUE/(B.VALUE+C.VALUE) RATIO FROM GV$SYSSTAT A, GV$SYSSTAT B, GV$SYSSTAT C WHERE A.NAME='global cache blocks lost' AND B.NAME='global cache current blocks served' AND C.NAME='global cache cr blocks served' and B.INST_ID=a.inst_id AND C.INST_ID = a.inst_id;
In this case the database Instance 1 takes 22 seconds to perform a series of tests, Instance 2 takes 25 minutes.
Instance gc blocks lost gc cur blocks served gc cr blocks served RATIO ---------- -------------- -------------------- ------------------- ---------1 0 3923 2734 0 2 652 3008 4380 .088251218
A TUSC Presentation
31
The TCP receive and send buffers on Instance 2 were set at 64K This is a 8k block size instance with a db_file_multiblock_read_count of 16. This was causing excessive network traffic since the system was using full table scans resulting in a read of 128K. In addition the actual TCP buffer area was set to a small number.
Tuning the RAC Cluster Interconnect Current Block Transfer Statistics
In addition to monitoring consistent read blocks, we also need to be concerned with processing current mode blocks. Calculate the average receive time for current mode blocks:
Global cache current block receive time Global cache current blocks received
Accumulated round trip time for all requests for current blocks
Number of current blocks received from the holding instance over the interconnect
Tuning the RAC Cluster Interconnect Current Block Transfer Statistics
Query the GV$SYSSTAT view to obtain this ratio for each instance:
COLUMN COLUMN PROMPT SELECT "AVG RECEIVE TIME (ms)" format 9999999.9 inst_id FORMAT 9999 GCS CURRENT BLOCKS b1.inst_id, b2.value "RECEIVED", b1.value "RECEIVE TIME", ((b1.value / b2.value) * 10) "AVG RECEIVE TIME (ms)" gv$sysstat b1, gv$sysstat b2 b1.name = 'global cache current block receive time' b2.name = 'global cache current blocks received' b1.inst_id = b2.inst_id;
FROM WHERE AND AND
INST_ID RECEIVED RECEIVE TIME AVG RECEIVE TIME (ms) ------ ---------- ------------ --------------------1 22694 68999 30.4 2 23931 42090 17.6 A TUSC Presentation 34
Tuning the RAC Cluster Interconnect Current Block Service Time Statistics
Service time for current blocks is comprised of pin time, log flush time, and send time.
global cache current blocks served
The number of current blocks shipped to the requesting instance over the interconnect.
global cache block pin time
The time it takes to pin the current block before shipping it to the requesting instance. Pinning is necessary to disallow changes to the block while it is prepared to be shipped to another instance.
global cache block flush The time it takes to flush the changes to a block to disk (forced log flush), before the block is shipped time
to the requesting instance.
global cache block send time
The time it takes to send the current block to the requesting instance over the interconnect.
To calculate current block service time, query GV$SYSSTAT:

SELECT a.inst_id "Instance", (a.value+b.value+c.value)/d.value "Current Blk Service Time" FROM GV$SYSSTAT A, GV$SYSSTAT B, GV$SYSSTAT C, GV$SYSSTAT D WHERE A.name = 'global cache current block pin time' AND B.name = 'global cache current block flush time' AND C.name = 'global cache current block send time' AND D.name = 'global cache current blocks served' AND B.inst_id = A.inst_id AND C.inst_id = A.inst_id AND D.inst_id = A.inst_id ORDER BY a.inst_id;
A TUSC Presentation
36
The output from the query may look like this:

Instance Current Blk Service Time --------- -----------------------1 1.18461603 2 1.63126376 Instance 2 requires 38% more time to service current blocks than Instance 1.
Drill-down to service time components will help identify the cause of the problem.
A TUSC Presentation
37
SELECT A.inst_id "Instance", (A.value/D.value) "Current Block Pin", (B.value/D.value) "Log Flush Wait", (C.value/D.value) "Send Time" FROM GV$SYSSTAT A, GV$SYSSTAT B, GV$SYSSTAT C, GV$SYSSTAT D WHERE A.name = 'global cache current block build time' AND B.name = 'global cache current block flush time' AND C.name = 'global cache current block send time' AND D.name = 'global cache current blocks served' AND B.inst_id=a.inst_id AND High pin times could indicate AND C.inst_id=a.inst_id AND AND D.inst_id=a.inst_id problems at the IO interface level. ORDER BY A.inst_id;
Instance Current Block Pin Log Flush Wait Send Time --------- ----------------- -------------- ---------1 .69366887 .472058762 .018196236 2 1.07740715 .480549199 .072346418
Tuning the RAC Cluster Interconnect Global Cache Convert and Get Times
GCS: Global Cache Services. A process that communicates through the cluster interconnect
A final set of statistics can be useful in identifying RAC interconnect performance issues.
global cache convert time global cache converts
The accumulated time that all sessions require to perform global conversion on GCS resources Resource converts on buffer cache blocks. Incremented whenever GCS resources are converted from Null to Exclusive, shared to Exclusive, or Null to Shared.
global cache get time

Global cache gets
The accumulated time of all sessions needed to open a GCS resource for a local buffer.
The number of buffer gets that result in opening a new resource with the GCS.
To measure these times, query the GV$SYSSTAT view:
SELECT A.inst_id "Instance", A.value/B.value "Avg Cache Conv. Time", C.value/D.value "Avg Cache Get Time", E.value "GC Convert Timeouts" FROM GV$SYSSTAT A, GV$SYSSTAT B, GV$SYSSTAT C, GV$SYSSTAT D, Neither convert time is excessive. GV$SYSSTAT Excessive would be > 10-20 ms. Instance WHERE A.name = 'global cache convert time' 1 has higher convert times because it is AND B.name = 'global cache converts' getting and converting from instance 2, AND c.name = 'global cache get time' which is running on slower CPUs AND D.name = 'global cache gets' AND E.name = 'global cache convert timeouts' AND B.inst_id = A.inst_id and AND C.inst_id = A.inst_id and AND D.inst_id = A.inst_id and AND E.inst_id = A.inst_id ORDER BY A.inst_id;
Instance Avg Cache Conv. Time Avg Cache Get Time GC Convert Timeouts --------- -------------------- ------------------ ------------------1 1.85812072 .981296356 0 2 1.65947528 .627444273 0
Use the GV$SYSTEM_EVENT view to review TIME_WAITED statistics for various GCS events if the get or convert times become significant. STATSPACK can be used for this to view these events.
A TUSC Presentation
41
Interpreting the convert and get statistics

High convert times Large values or rapid increases in gets, converts or average times High latencies for resource operations Non-zero GC converts Timeouts
Instances swapping a lot of blocks over the interconnect GCS contention Excessive system loads System contention or congestion. Could indicate serious performance problems.
42
A TUSC Presentation
Tuning the RAC Cluster Interconnect Other Wait Events
If these RAC-specific wait events show up in the TOP-5 wait events on the STATSPACK report, you need to determine the cause of the waits:
global cache open s global cache open x global cache null to s global cache null to x global cache cr request Global Cache Service Utilization for Logical Reads
Event: global cache open s global cache open x
Description: A session has to wait for receiving permission for shared(s) or exclusive(x) access to the requested resource Wait duration should be short. Wait followed by a read from disk. Blocks requested are not cached in any instance.
Action: Not much can be done When associated with high totals or high per-transaction wait time, data blocks are not cached. Will cause sub-optimal buffer cache hit ratios Consider preloading heavily used tables
Event: global cache null to s global cache null to x
Description: Happens when two instances exchange the same block back and forth over the network. These events will consume a greater proportion of total wait time if one instance requests cached data blocks from other instances.
Action: Reduce the number of rows per block to eliminate the need for block swapping between instances.
Event: global cache cr request
Description: Happens when an instance requests a CR data block and the block to be transferred hasnt arrived at the requesting instance.
Action: Examine the cluster interconnects for possible problems. Modify objects to reduce possibility of contention.
Tuning the RAC Cluster Interconnect High GCS Time Per Request
Examine the Cluster Statistics page of your STATSPACK report when:

Global cache waits constitute a large proportion of the wait time listed on the first page of your STATSPACK report.
AND
Response times or throughput does not conform to your service level requirements.
STATSPACK report should be taken during heavy RAC workloads.

Causes of a high GCS time per request are:
Contention for blocks System Load Network issues
A TUSC Presentation
48
Contention for blocks You already know this:

Modify the object to reduce the chances for application contention. Reduce the number of rows per block Adjust the block size to a smaller block size Modify INITRANS and FREELISTS
A TUSC Presentation
49
System Load
If processes are queuing for the CPU, raise the priority of the GCS processes (LMSn) to have priority over other processes to lower GCS times. Reduce the load on the server by reducing the number of processes on the database server. Increase capacity by adding CPUs to the server Add nodes to the cluster database
Network issues
OS logs and OS stats will indicate if a network link is congested. Ensure packets are being routed through the private interconnect, not the public network.
A TUSC Presentation
51
Tuning the RAC Cluster Interconnect Using STATSPACK Reports
The STATSPACK report shows statistics ONLY for the node or instance on which it was run. Run statspack.snap procedure and spreport.sql script on each node you want to monitor to compare to other instances.
A TUSC Presentation
52

Exceeds CPU time, therefore needs investigation.
Top 5 Timed Events
CPU time (processing time) should be the predominant event
Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Ela Time -------------------------------------------- ------------ ----------- -------global cache cr request 820 154 72.50 CPU time 54 25.34 global cache null to x 478 1 .52 control file sequential read 600 1 .52 control file parallel write 141 1 .28 -------------------------------------------------------------
Transfer times are excessive from other instances in this cluster to this instance. Could be due to network problems or buffer cache sizing issues.
Network changes were made An index was added STATSPACK report now looks like this:
CPU time is now the predominant event
Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Ela Time -------------------------------------------- ------------ ----------- -------CPU time 99 64.87 global cache null to x 1,655 28 18.43 enqueue 46 8 5.12 global cache busy 104 7 4.73 DFS lock handle 38 2 1.64
A TUSC Presentation
54
Tuning the RAC Cluster Interconnect After network and index Using STATSPACK Reports changes.
Workload characteristics for this instance:
Cluster Statistics for DB: DB2 Instance: INST1 Snaps: 105 -106 Snaps: 25 -26 Global Cache Service - Workload Characteristics ----------------------------------------------Ave global cache get time (ms): Ave global cache convert time (ms): Ave build time for CR block (ms): Ave flush time for CR block (ms): Ave send time for CR block (ms): Ave time to process CR block request (ms): Ave receive time for CR block (ms): Ave pin time for current block (ms): Ave flush time for current block (ms): Ave send time for current block (ms): Ave time to process current block request (ms): Ave receive time for current block (ms): Global cache hit ratio: Ratio of current block defers: % of messages sent for buffer gets: % of remote buffer gets: Ratio of I/O for coherence: Ratio of local vs remote work: Ratio of fusion vs physical writes: A TUSC Presentation
3.1 3.2 0.2 0.0 1.0 1.3 17.2 0.2 0.0 0.9 1.1 3.1 1.7 0.0 1.4 1.1 8.7 0.6 1.0
8.2 16.5 1.5 6.0 0.9 8.5 18.3 13.7 3.9 0.8 18.4 17.4 2.5 0.2 2.2 1.6 2.9 0.5 0.0 55
Global Enqueue Services (GES) control the interinstance locks in Oracle 9i RAC. The STATSPACK report contains a special section for these statistics.
Global Enqueue Service Statistics --------------------------------Ave global lock get time (ms): Ave global lock convert time (ms): Ratio of global lock gets vs global lock releases:
0.9 1.3 1.1
A TUSC Presentation
56
Guidelines for GES Statistics:

All times should be < 15ms Ratio of global lock gets vs global lock releases should be near 1.0
High values could indicate possible network or memory problems Could also be caused by application locking issues May need to review the enqueue section of STATSPACK report for further analysis.
GCS AND GES messaging

Watch for excessive queue times (>20-30ms)
GCS and GES Messaging statistics -------------------------------Ave message sent queue time (ms): Ave message sent queue time on ksxp (ms): Ave message received queue time (ms): Ave GCS message process time (ms): Ave GES message process time (ms): % of direct sent messages: % of indirect sent messages: % of flow controlled messages:
1.8 2.6 1.2 1.2 0.2 58.4 4.9 36.7
A TUSC Presentation
58
The GES Statistics section of the STATSPACK report shows gcs blocked convert statistics.
GES Statistics for DB: DB2 Instance: INST2 Snaps: 25 -26 Statistic Total per Second per Trans --------------------------------- ---------------- ------------ -----------dynamically allocated gcs resource 0 0.0 0.0 dynamically allocated gcs shadows 0 0.0 0.0 flow control messages received 0 0.0 0.0 flow control messages sent 0 0.0 0.0 gcs ast xid 0 0.0 0.0 gcs blocked converts 904 2.1 150.7 gcs blocked cr converts 1,284 2.9 214.0 gcs compatible basts 0 0.0 0.0 gcs compatible cr basts (global) 0 0.0 0.0 gcs compatible cr basts (local) 75 0.2 12.5 : :
Statistic: gcs blocked converts gcs blocked cr converts
Description: Instance requested a block from another instance and was unable to obtain the conversion of the block. Indicates users on different instances need to access the same blocks.
Action: Prevent users on different instances from needing access to the same blocks. Ensure sufficient freelists (not an issue if using automated freelists) Reduce block contention through freelists, initrans. Limit rows per block.
Waits are a key tuning indicator:

Wait Events for DB: DB2 Instance: INST2 Snaps: 25 -26 -> s - second -> cs - centisecond 100th of a second -> ms - millisecond 1000th of a second -> us - microsecond - 1000000th of a second -> ordered by wait time desc, waits desc (idle events last) Our predominate wait. Network issues Indicates potential problems with the IO subsystem if severe. (not severe here, total wait time is < 1 sec)
Event Waits Timeouts Time (s) (ms) /txn ---------------------------- ------------ ---------- ---------- ------ -------global cache cr request 820 113 154 188 136.7 global cache null to x 478 1 1 2 79.7 control file sequential read 600 0 1 2 100.0 control file parallel write 141 0 1 4 23.5 enqueue 29 0 1 18 4.8 library cache lock 215 0 0 2 35.8 db file sequential read 28 0 0 7 4.7 LGWR wait for redo copy 31 16 0 4 5.2 : : A TUSC Presentation :
61
The Library Cache Activity report shows statistics regarding the GES. Watch for GES Invalid Requests and GES Invalidations. Could indicate insufficient sizing of the shared pool resulting in GES contention.
Library Cache Activity for DB: DB2 ->"Pct Misses" should be very low Instance: INST2 Snaps: 25 -26
GES Lock GES Pin GES Pin GES Inval GES InvaliNamespace Requests Requests Releases Requests dations --------------- ------------ ------------ ------------ ----------- ----------BODY 1 0 0 0 0 CLUSTER 4 0 0 0 0 INDEX 84 0 0 0 0 SQL AREA 0 0 0 0 0 TABLE/PROCEDURE 617 192 0 77 0 TRIGGER 0 0 0 0 0
A TUSC Presentation
62
Single-instance tuning should be performed before attempting to tune the processes that communicate via the cluster interconnect.
The STATSPACK report contains valuable statistics that can help you identify SQL-related problems, memory problems, IO problems, latch contention and enqueue issues.
A TUSC Presentation
63
Tuning the RAC Cluster Interconnect GES Monitoring
Oracle makes a Global Cache Service request whenever a user accesses a buffer cache to read or modify a data block and the block is not in the local cache. and the
crowd gasps!
RATIOS can be used to give an indication of how hard your Global Services Directory processes are working.
A TUSC Presentation
64
To estimate the use of the GCS relative to the number of logical reads: Sum of GCS requests
global cache gets + global cache converts + global cache cr blocks rcvd + global cache current blocks rcvd consistent gets + db block gets
Number of logical reads
A TUSC Presentation
65

This information can be found by querying the GV$SYSSTAT view
SELECT a.inst_id "Instance", (A.VALUE+B.VALUE+C.VALUE+D.VALUE)/(E.VALUE+F.VALUE) "GLOBAL CACHE HIT RATIO" FROM GV$SYSSTAT A, GV$SYSSTAT B, GV$SYSSTAT C, GV$SYSSTAT D, GV$SYSSTAT E, GV$SYSSTAT F WHERE A.NAME='global cache gets' AND B.NAME='global cache converts' AND C.NAME='global cache cr blocks received' AND D.NAME='global cache current blocks received' SINCE INSTANCE AND E.NAME='consistent gets' STARTUP AND F.NAME='db block gets' AND B.INST_ID=A.INST_ID AND C.INST_ID=A.INST_ID AND D.INST_ID=A.INST_ID AND E.INST_ID=A.INST_ID AND F.INST_ID=A.INST_ID;
Instance GLOBAL CACHE HIT RATIO ---------- ---------------------1 .02403656 2 .014798887

Some blocks, those frequently requested by local and remote users, will be hot. If a block is hot, its transfer is delayed for a few milliseconds to allow the local users to complete their work. The following ratio provides a rough estimate of how prevalent this is:
global cache defers
global cache current blocks served
A ratio of higher than 0.3 indicates that you have some pretty hot blocks in your database.
To find the blocks involved in busy waits, query the columns NAME, KIND, FORCED_READS, FORCED_WRITES.
Select INST_ID "Instance", name, kind, sum(forced_reads) "Forced Reads", sum(forced_writes) "Forced Writes" FROM gv$cache_transfer WHERE owner# != 0 GROUP BY inst_id, name, kind ORDER BY 1,4 desc,2;
Instance NAME KIND Forced Reads Forced Writes --------- -------------------- ---------- ------------ ------------1 MOD_TEST_IND INDEX 308 0 1 TEST2 TABLE 64 0 1 AQ$_QUEUE_TABLES TABLE 5 0 2 TEST2 TABLE 473 0 2 MOD_TEST_IND INDEX 221 0
If you discover a problem, you may be able to alleviate contention by:

Reducing hot spots or spreading the accesses to index blocks or data blocks. Use Oracle has or range partitions wherever applicable, just as you would in single-instance Oracle databases Reduce concurrency on the object by implementing resource management or load balancing. Decrease the rate of modifications on the object (use fewer database processes).
Fusion writes occur when a block previously changed by another instance needs to be written to disk in response to a checkpoint or cache aging. Oracle sends a message to notify the other instance that a fusion write will be performed to move the data block to disk. Fusion writes do not require an additional write to disk. Fusion writes are a subset of all physical writes incurred by an instance.

The following ratio shows the proportion of writes that Oracle manages with fusion writes:
DBWR fusion writes physical writes
SELECT A.inst_id "Instance", A.VALUE/B.VALUE "Cache Fusion Writes Ratio" FROM GV$SYSSTAT A, GV$SYSSTAT B WHERE A.name='DBWR fusion writes' AND B.name='physical writes' AND B.inst_id=a.inst_id ORDER BY A.INST_ID;
Instance Cache Fusion Writes Ratio --------- ------------------------1 .216290958 2 .131862042

A high large value for Cache Fusion Writes ratio may indicate:
Insufficiently sized caches Insufficient checkpoints Large numbers of buffers written due to cache replacement or checkpointing.
A TUSC Presentation
72
Tuning the RAC Cluster Interconnect CACHE_TRANSFER views
The V$XXX_CACHE_TRANSFER views show the types of blocks that use the cluster interconnect in a RAC environment.
A TUSC Presentation
73
V$CACHE_TRANSFER
Shows the types and classes of blocks that Oracle transfers over the cluster interconnect on a per-object basis. that experience cache transfers on a per-class of object basis. Use this view to monitor the blocks transferred per file. The FILE_NUMBER column identifies the datafile that contained the blocks transferred. Use this view to monitor the transfer of temporary tablespace blocks. Has the same structure as V$FILE_CACHE_TRANSFER.
V$CLASS_CACHE_TRANSFER Use this view to identify the class of blocks
V$FILE_CACHE_TRANSFER
V$TEMP_CACHE_TRANSFER

FORCED_READS and FORCED_WRITES column are used to determine which types of objects your RAC instances are sharing. Values in FORCED_WRITES column provide counts of how often a certain block type experiences a transfer out of a local buffer cache due to a request for current version by another instance. The NAME column shows the name of the object containing blocks being transferred.
V$CACHE_TRANSFER
FILE# BLOCK# CLASS# STATUS XNC FORCED_READS FORCED_WRITES NAME PARTITION_NAME KIND OWNER# GC_ELEMENT_ADDR GC_ELEMENT_NAME NUMBER NUMBER NUMBER VARCHAR2(5) NUMBER NUMBER NUMBER VARCHAR2(30) VARCHAR2(30) VARCHAR2(15) NUMBER RAW(4) NUMBER
A TUSC Presentation
75
V$FILE_CACHE_TRANSFER FILE_NUMBER shows the file numbers of datafiles that are source of blocks transferred. Use this view to show the block transfers per file.
FILE_NUMBER X_2_NULL X_2_NULL_FORCED_WRITE X_2_NULL_FORCED_STALE X_2_S X_2_S_FORCED_WRITE S_2_NULL S_2_NULL_FORCED_STALE RBR RBR_FORCED_WRITE RBR_FORCED_STALE NULL_2_X S_2_X NULL_2_S CR_TRANSFERS CUR_TRANSFERS NUMBER NUMBER NUMBER NUMBER NUMBER NUMBER NUMBER NUMBER NUMBER NUMBER NUMBER NUMBER NUMBER NUMBER NUMBER NUMBER
76
A TUSC Presentation
The shared disk architecture of Oracle 9i virtually eliminates forced disk writes, but V$CACHE_TRANSFER and V$FILE_CACHE_TRANSFER may still show the number of block mode conversions per block class or object, however, the values in FORCED_WRITES column will be zero.
A TUSC Presentation
77
Tuning the RAC Cluster Interconnect Monitoring the GES Processes
To monitor the Global Enqueue Service Processes, use the GV$ENQUEUE_STAT view.
GV$ENQUEUE_STAT
INST_ID EQ_TYPE TOTAL_REQ# TOTAL_WAIT SUCC_REQ# FAILED_REQ# CUM_WAIT_TIME NUMBER VARCHAR2(2) NUMBER NUMBER NUMBER NUMBER NUMBER
A TUSC Presentation
78
Retrieve all of the enqueues with a total_wait# value greater than zero:
SELECT * FROM gv$enqueue_stat WHERE total_wait# > 0 ORDER BY inst_id, cum_wait_time desc; INST_ID ---------1 1 1 1 1 1 1 : : EQ TOTAL_REQ# TOTAL_WAIT# SUCC_REQ# FAILED_REQ# CUM_WAIT_TIME -- ---------- ----------- ---------- ----------- ------------TX 31928 26 31928 0 293303 PS 995 571 994 1 55658 TA 1067 874 1067 0 10466 TD 974 974 974 0 2980 DR 176 176 176 0 406 US 190 189 190 0 404 PI 47 27 47 0 104
Oracle says enqueues of interest in the RAC environment are:

SQ Enqueue
Indicates there is contention for sequences. Increase the cache size of sequences using ALTER SEQUENCES. Also, when creating sequences, the NOORDER keyword should be used to avoid forced ordering of queued sequence values. processing can magnify the effect of TX enqueue waits. Index block splits can cause TX enqueue waits. Some TX enqueue performance issues can be resolved by setting the value of INITRANS parameter for a table or index (have heard to set equal to the number of CPUs per node multiplied by the number of nodes in a cluster multiplied by 0.75). I prefer setting INITTRANS to the number of concurrent processes issuing DML against the block.
TX Enqueue Application-related row locking usually causes problems here. RAC
Enterprise Manager 9i
A TUSC Presentation
81
Enterprise Manager 9i
A TUSC Presentation
82
Statspack Top Wait Events
Top 5 Timed Events ~~~~~~~~~~~~~~~~~~

Event Waits Time (s) --------------------------- ------------ ----------db file sequential read 399,394,399 2,562,115 CPU time 960,825 buffer busy waits 122,302,412 540,757 PL/SQL lock timer 4,077 243,056 log file switch 188,701 187,648 (checkpoint incomplete) % Total Ela Time -------52.26 19.60 11.03 4.96 3.83
A TUSC Presentation
83
Statspack - Top Wait Events Things to look for

Wait Problem Potential Fix Sequential Read Indicates many index reads tune the code (especially joins) Scattered Read Indicates many full table scans tune the code; cache small tables Free Buffer Increase the DB_CACHE_SIZE; shorten the checkpoint; tune the code Buffer Busy Segment Header Add freelists or freelist groups (Use ASSM) Buffer Busy Data Block Separate hot data; use reverseA key indexes; smaller blocks84 TUSC Presentation

Wait Problem Buffer Busy
Buffer Busy Buffer Busy
Latch Free Enqueue - ST Enqueue - HW
Potential Fix Data Block (cont.) Increase initrans and/or maxtrans Undo Header Add rollback segments or areas Undo block Commit more (not too much) Larger rollback segments/areas. Investigate the detail (Covered later) Use LMTs or pre-allocate large extents Pre-allocate extents above high water mark A TUSC Presentation 85

Wait Problem Enqueue TX (transaction) Enqueue - TM (trans. mgmt.) Log Buffer Space Log File Switch
Log file sync
Potential Fix Increase initrans and/or maxtrans on the table or index Index foreign keys; Check application locking of tables Increase the Log Buffer; Faster disks for the Redo Logs Archive destination slow or full; Add more or larger Redo Logs Commit more records at a time; Faster Redo Log disks; Raw devices A TUSC Presentation 86
Statspack - Latch Waits Things to look for

Latch Problem Library Cache Shared Pool Row cache objects Redo allocation Redo copy Potential Fix Use bind variables; adjust the shared_pool_size Use bind variables; adjust the shared_pool_size Increase the Shared Pool Minimize redo generation and avoid unnecessary commits Increase the _log_simultaneous_copies
Locally Managed Tablespaces- LMT

Manage space locally using bitmaps Benefits
no tablespace fragmentation issues better performance handling on large segments lower contention for central resources, e.g., no ST enqueue contention fewer recursive calls
Implementation
specify EXTENT MANAGEMENT LOCAL clause during tablespace creation in-place migration
Automatic Segment Space Management (ASSM)

Automatic intra-object space management Benefits
Eliminates Rollback Segment Management simplified administration (no more FREELISTS, FREELIST GROUPS, PCTUSED) improved space utilization & concurrency enhanced performance
Can set undo retention time

If set to longest running query time, no more Snapshot too old!
Implementation
specify SEGMENT SPACE MANAGEMENT AUTO A TUSC Presentation 89 clause during tablespace creation
Enterprise Manager 10g for the Grid
Host and Hardware
Database Oracle9iAS
Network and Load Balancer
Administration Monitoring Provisioning Security

Enterprise Manager
Applications
Storage
A TUSC Presentation
90
Whats New in EM 10g

Area
Oracle Database Oracle9iAS Oracle Collab Suite Oracle eBusiness Suite Operating System Storage Network SLB (Nortel, F5) Hardware Admin and Monitoring
A TUSC Presentation
EM 9i
EM 10g
Key EM10G functionality

Application Performance Management
Real performance for All Your Users, All Your Pages, All the Time Application Availability Synthetic transactions End-to-end tracing
Enterprise Configuration Management

Rapid installation Deployment Provisioning Upgrade Automated patching
91
End to End Solution

DB iAS
Config changes across cluster (OC4J, Apache) Create/Manage cluster Deploy app to cluster Reconfigure a farm Add/Remove node from cluster Clone (mid-tiers) Patch General health assessment Bad SQL identification Top SQL identification SQL recommendations Tuning advisories Performance trending Backup Restore Security vulnerability ID Create/remove Physical/Logical Standby Standby health assessment Standby switchover/failover
All Target Types

System and application availability Set up of target-specific metrics/thresholds Configuration data collection Performance data collection Comprehensive monitoring Alerts Email and Paging notifications Blackouts User-defined jobs System and application response time measurements Policy violation reporting Clone Oracle Home Patch search/download
OCS
IMAP, SMTP End-2-end service monitoring Files End-2-end service monitoring Files document analysis: count, size, format Files user analysis: number, quota consumed
RAC
Cluster cache coherency monitoring Failover events Discovery of RAC topology Startup/shutdown Relocate services Failover jobs
A TUSC Presentation
ASM
Disk group admin (rebalance) Startup/Shutdown Disk group usage/status I/O performance 92 Add/Remove disks
Host/Host Clusters
Top processes identification
Monitor All Targets

Monitor Targets are they up?
Target Alerts
A TUSC Presentation
93
View it Wireless
Target Alerts
A TUSC Presentation
94
Application Performance Issues
Internet
Application Developer Applications Application Server Data Center Support Database OS Storage Network Service Network DBA System Administrator
Who is the problem impacting? What is the business impact? Where in the infrastructure is the problem? Whose problem is it? Why is the problem occurring?
95
Provider
A TUSC Presentation
Middle Tier Performance Analysis

Application server and back-end problem diagnostics Middle tier processing time and load breakouts Find slowest URLs and get number of hits Top servlets / JSPs by requests /processing time Tuning recommendations
Monitor App. Servers

Monitor Web Apps.
Web App. Alerts
A TUSC Presentation
97
Choose a Host
Select Targets tab Pick a Host
A TUSC Presentation
98
App. Server Response Time
Choose App. Server
App Server Response Time & Info.

App. Server Performance

Memory CPU
Web Cache Hits

View the Web Application

Choose Web Apps. Choose the App. To Monitor
A TUSC Presentation
101

Page Response Time Detailed Info.
Click Alert for this
A TUSC Presentation
102

Page Response Time Detail
A TUSC Presentation
103
URL Response Times

Web Page Response Time all URLs
A TUSC Presentation
104
Middle Tier Performance

Full URL processing call stack analysis Middle tier processing time and load breakouts Tracing down to the SQL statement level
Servlet JSP EJB
JDBC
internet
Middle Tier A TUSC Presentation
105
Middle Tier Performance

Web Page Detail Response Time all URLs
Splits Time into Parts
A TUSC Presentation
106
Host Performance
Each Piece of the Web App. Choose the Host
A TUSC Presentation
107
Host Performance
I/O Memory CPU
A TUSC Presentation
108
Host Performance
Hardware
O/S Software
A TUSC Presentation
109
Host Performance
Side by Side Compare
A TUSC Presentation
110
Managing Groups
Managed from a single-view
Monitoring and automated operations
Applications
Logical modeling of sets of systems

Applications, Clusters, other sets Leveraged by all services Jobs, Policies,
Sets of Systems
Membership-based inheritance
Automated Group Operations

All Target Types
Config. change tracking Configuration inventory Diff configurations Search configurations Behavior inheritance (Jobs) Create/manage groups OS command jobs SQL Script jobs Aggregate Metrics Alert Rollups Set blackouts Set monitoring levels Set target properties Set thresholds Installation hardening Security alerts and patches Discovery
ASM
Disk group admin (rebalance) Startup/Shutdown Disk group usage/status I/O performance Add/Remove disks
iAS
Config changes across cluster (OC4J, Apache) Create/Manage cluster Deploy app to cluster Reconfigure a farm Add/Remove node from cluster Clone (mid-tiers) Patch
DB
Analyze Backup Export Startup/Shutdown Configuration Advise Clone, Patch
OCS
Custom grouping Home pages for EMAIL and IM
RAC
WebApp
End-2-End availability End-2-End monitoring End-2-End tracing
Host/Host Clusters
Add/Remove node from OS cluster
Spfile changes across instances Start/Stop/Relocate services EM Startup/Shutdown Cluster cache coherency Agent deployment Monitoring rollups A TUSC Presentation Add/Remove Instance
112
Database Performance
Create a Prod DBs Group
Pick Items for the Group

View the Prod DBs Group
Check Alerts & Policy Issues
A TUSC Presentation
114
Alert History Waits
Advice Click Here for Issues

Alert Issues Major Waits
A TUSC Presentation
116
Fix
Critical
A TUSC Presentation
117
Oracle Standardization
Policy Management
Policy
Rule definitions Violation detection Corrective action
Security policies
Software installation hardening Excess services/ports Excess user privileges
Configuration policies
Best practices Base images
Performance polices
Thresholds
Automated Best Practices Database

1. Insufficient Number of Control Files 2. Insufficient Redo Log Size 11. Not Using Locally Managed Tablespaces 12. SYSTEM TS Contains Non-System Data Seg 13. Users with Permanent TS as Temporary TS 14. Insufficient Recovery Area Size 15. Force Logging Disabled 16. Not Using Spfile 17. Rollback in SYSTEM Tablespace 18. Not Using Undo Space Management
3. Insufficient Number of Redo Logs

4. Use of Unlimited Autoextension 5. Use of Non-Standard Init. Parameters 6. Recovery Area Location Not Set
7. Autobackup of Control File is not Enabled

8. SYSTEM TS Used as User Default TS
9. Segment with Extent Growth Policy Violation 19. Non-uniform Default Extent Size 10.Tablespace Containing Mixed Segment Types
A TUSC Presentation
119
Automated Security Checks

All Oracle Software
1. 2. 1. 2. Security alerts Critical patches Detect open ports Detect insecure services
Database Services
1. 2. 3. 4. 5. 6. Enable listener logging Password-protect listeners Disable direct listener administration Disallow remote OS roles and authentication Disallow use of remote password file Restrict access to external procedure service
Host
Application Server
1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13.
Database User Privileges

Disable install and demo accounts Disallow default user/password PUBLIC has execute System privilege PUBLIC has execute Object privilege PUBLIC has execute UTL_FILE privilege PUBLIC has execute UTL_SMTP privilege PUBLIC has execute UTL_HTTP privilege PUBLIC has execute UTL_TCP privilege PUBLIC has execute DBMS_RANDOM Password complexity Restrict number of failed login attempts Authentication protocol fallback 120 Connect and Resource grants
HTTPD has minimal privileges 1. Use HTTP/S 2. Apache logging should be on 3. Demo applications disabled 4. Disable default banner page 5. Disable access to unused directories 6. Disable directory indexing 7. Forbid access to certain packages 8. Disable packages not used by DAD owner 9. Remove unused DAD configurations 10. Redirect _pages directory 11. Password complexity enabled 12. A TUSC Presentation Use HTTP/S 13.
Monitor Database We have a CPU issue!
A TUSC Presentation
121
Monitor Perform. We have a Concurr. issue!
A TUSC Presentation
122
Monitor for a period of time Check the Top SQL
A TUSC Presentation
123
Drill down to show the Top SQL
Target Alerts
Top SQL Check Execution History or Other Options
A TUSC Presentation
125
Use Automatic Database Diagnostics Monitor (ADDM)

Monitor Database Check the Diagnostic Summary Target Alerts

Check a snapshot Check the findings
A TUSC Presentation
127
Finding Details Suggests!
A TUSC Presentation
128

I can do it for you !
ADDM
High-Load SQL
DBA
SQL Workload SQL Tuning Advisor
A TUSC Presentation
129
ADDMs Architecture
Uses Time & Wait Model data from Workload Repository Classification Tree is based on decades of Oracle performance tuning expertise Time based analysis Recommends solutions or next steps Runs proactively & manually
Snapshots in Automatic Workload Repository
Automatic Diagnostic Engine Automatic Diagnostic Engine
High-load SQL
IO / CPU issues
RAC issues
SQL Advisor
System Sizing Advice
Network + DB config Advice A TUSC Presentation
130
Automatic Workload Repository

Like a better statspack! Repository of performance information
Base statistics SQL statistics Metrics ACTIVE SESSION HISTORY
Workload data is captured every 30 minutes or manually and saved for 7 days by default Self-manages its space requirements Provides the basis for improved performance diagnostic facilities
A TUSC Presentation
131
Automatic Tuning Optimizer (ATO)
Automatic Tuning Optimizer

Statistics Analysis
SQL Profiling
SQL Tuning Recommendations SQL Tuning Advisor

Gather Missing or Stale Statistics
Create a SQL Profile

DBA
Access Path Analysis
Add Missing Indexes
SQL Structure Analysis

A TUSC Presentation
Modify SQL Constructs

132
Automatic Tuning Optimizer (ATO)
It is the query optimizer running in tuning mode
Uses same plan generation process but performs additional steps that require lot more time To validate statistics and its own estimates
Uses dynamic sampling and partial executions
It performs verification steps
It performs exploratory steps
To investigate the use of new indexes that could provide significant speed-up To analyze SQL constructs that led to expensive plan operators A TUSC Presentation 133
ADDM SQL Tuning Advisor
Drill into Top SQL for worst time period Get Help!
A TUSC Presentation
134
Top SQL; Use SQL Tuning Sets
Tuning Sets Tune the Top SQL set.
A TUSC Presentation
135
Using SQL Tuning Sets
SQL Tuning Sets Info
How long to work on this
A TUSC Presentation
136
Tuning Results for this job Suggests using a profile For this SQL
Detailed info Improve Factor and click to Use
A TUSC Presentation
138
Confirmed to use suggestion
Detail
A TUSC Presentation
139
Using SQL Tuning Sets (STS)

Motivation
Enable user to tune a custom set of SQL statements
It is a new object in Oracle10g for capturing SQL workload It stores SQL statements along with

Execution context: parsing user, bind values, etc. Execution statistics: buffer gets, CPU time, elapse time, number of executions, etc.
It is created from a SQL source
Sources: AWR, cursor cache, user-defined SQL A TUSC Presentation 140 workload, another STS
Business Transaction Monitoring - Set up Beacons!

Monitor critical online business processes Scope and quantify impact of performance problems on all user communities Isolate network vs server related delays Transaction profiling Alerts and notifications
internet
Web Application
Transaction Monitoring Recorder

Rapid deployment and immediate value Simple, automatic, no scripting Beacons automatically synchronized to play transactions at specific intervals
Download Recorder (once) Start recorder which opens new browser window Enter URL and navigate through transaction Save Transaction
A TUSC Presentation
142
Performance Monitoring Topology

End Users Network Apps and Mid-Tier Servers Oracle E-Bus Suite Oracle9iAS Database Hosts Storage
Beacon running representative transaction
Beacon running representative transaction Beacon running representative transaction Beacon running representative transaction
Oracle9iAS
Beacon running representative transaction
Oracle Oracle9i Database Database
A TUSC Presentation
3rd Party App Server
143
Availability Monitoring Topology

End Users
Chicago Sales Office
Beacon running availability transaction
Network
Apps and Mid-Tier Servers Oracle E-Bus Suite

Oracle9iAS
Database
Hosts
Storage
Toronto Sales Office

Oracle9iAS
Oracle Database
Headquarters
Tokyo Sales Office
A TUSC Presentation
3rd Party App Server
144
Enterprise Manager for the Grid; Many more Options!
A TUSC Presentation
145
Summary
Tune each instance in a database cluster independently prior to tuning RAC. Reduce contention for blocks to improve service time. Operate on a well-tuned network. Monitor system load. Use V$ views to monitor RAC systems. STATSPACK contains vital information for RAC systems. New Features of Oracle10g will ease administration even further.
A TUSC Presentation
146
Questions and Answers
A TUSC Presentation
147
For More Information

www.tusc.com
Oracle9i Performance Tuning Tips & Techniques; Richard J. Niemiec; Oracle Press (May 2003)
If you are going through hell, keep going - Churchill
References
Oracle9i Performance Tuning Tips & Techniques, Rich
Niemiec
The Self-managing Database: Automatic Performance Diagnosis; Karl Dias & Mark Ramacher, Oracle
Corporation
EM Grid Control 10g; otn.oracle.com, Oracle Corporation Oracle Enterprise Manager 10g: Making the Grid a Reality; Jay Rossiter, Oracle Corporation The Self-Managing Database: Guided Application and SQL Tuning; Benoit Dageville, Oracle Corporation The New Enterprise Manager: End to End Performance 149 Management of OracleA;TUSC Presentation Julie Wong & Arsalan Farooq,
References
Oracle Database 10g Performance Overview; Herv
Lejeune, Oracle Corporation Oracle 10g; Penny Avril,, Oracle Corporation
Forrester Reports, Inc., TechStrategy Research, April

2002, Organic IT
Internals of Real Application Cluster, Madhu Tumma,

Credit Suisse First Boston
Oracle9i RAC; Real Application Clusters Configuration and Internals, Mike Ault & Madhu Tumma
References
Oracle Tuning Presentation, Oracle Corporation
www.tusc.com, www.oracle.com, www.ixora.com, www.laoug.org, www.ioug.org, technet.oracle.com
Oracle PL/SQL Tips and Techniques, Joseph P. Trezzo; Oracle Press Oracle9i Web Development, Bradley D. Brown; Oracle Press Special thanks to Steve Adams, Mike Ault, Brad Brown, Don Burleson, Kevin Gilpin, Herve Lejeune, Randy Swanson and Joe Trezzo.
References
Oracle Database 10g Automated Features , Mike Ault, TUSC Oracle Database 10g New Features, Mike Ault, Daniel Liu, Madhu Tumma, Rampant Technical Press, 2003, www.rampant.cc Oracle Database 10g - The World's First SelfManaging, Grid-Ready Database Arrives, Kelli Wiseth, Oracle Technology Network, 2003, otn.oracle.com
References
Oracle 10g documentation Oracle 9i RAC class & instructors comments Oracle 9i Concepts manual http://geocities.com/pulliamrick/ Tips for Tuning Oracle9i RAC on Linux, Kurt Engeleiter, Van Okamura, Oracle
Leveraging Oracle9i RAC on Intel-based servers to build an Adaptive Architecture, Stephen White,
Cap Gemini Ernst & Young, Dr Don Mowbray, Oracle, Werner Schueler, Intel
A TUSC Presentation
153
References
Running YOUR Applications on Real Application Clusters (RAC); RAC Deployment Best Practices,
Kirk McGowan, Oracle Corporation Oracle
The Present, The Future but not Science Fiction; Real Application Clusters Development, Angelo Pruscino, Building the Adaptive Enterprise; Adaptive Architecture and Oracle, Malcolm Carnegie, Cap
Gemini Ernst & Young
Internals of Real Application Cluster, Madhu Tumma,

Credit Suisse First Boston
A TUSC Presentation Creating Business Prosperity in a Challenging 154
References
Real Application Clusters, Real Customers Real Results, Erik Peterson, Technical Manager, RAC,
Oracle Corp.
Deploying a Highly Manageable Oracle9i Real Applications Database, Bill Kehoe, Oracle Getting the most out of your database, Andy
Mendelsohn, SVP Server Technologies, Oracle Corporation
Oracle9iAS Clusters: Solutions for Scalability and Availability, Chet Fryjoff, Product Manager, Oracle Corporation Oracle RAC and Linux A TUSC Presentation enterprise, Mark 155 in the real
References
Oracle Database 10g Automated Features , Mike Ault, TUSC Oracle Database 10g New Features, Mike Ault, Daniel Liu, Madhu Tumma, Rampant Technical Press, 2003, www.rampant.cc Oracle Database 10g - The World's First SelfManaging, Grid-Ready Database Arrives, Kelli Wiseth, Oracle Technology Network, 2003, otn.oracle.com
References
Oracle 10g; Penny Avril, Principal Database Product Manager, Server Technologies, Oracle Corporation To Infinity and Beyond, Brad Brown, TUSC Forrester Reports, Inc., TechStrategy Research, April 2002, Organic IT Internals of Real Application Cluster, Madhu Tumma, Credit Suisse First Boston
Oracle9i RAC; Real Application Clusters Configuration and Internals, Mike Ault & Madhu Tumma Oracle9i Performance Tuning Tips & Techniques,
Richard J. Niemiec
References
The Traits of the Uncommon Leader; U.S. Marine Corps Manual 60 Minute Manager - Joe Trezzo, TUSC Uncommon Leaders; TUSC, 1989 Whats next for IT?; Larry Geisel, Netscape The Miracle of Motivation; George Shinn Gods little devotional book for leaders 7 Habits of Highly Effective People; Steven Covey The Laws of Leadership - John Maxwell The making of a leader - Frank Damazio Bullet Proof Manager Seminars, Krestcom Productions, Inc.
TUSC Services
Oracle Technical Solutions
Full-Life Cycle Development Projects Enterprise Architecture Database Services
Oracle Application Solutions

Oracle Applications Implementations/Upgrades Oracle Applications Tuning
Managed Services
24x7x365 Remote Monitoring & Management Functional & Technical Support
Training & Mentoring

Copyright Information
Neither TUSC, Oracle nor the authors guarantee this document to be error-free. Please provide comments/questions to rich@tusc.com. TUSC 2004. This document cannot be reproduced without expressed written consent from an officer of TUSC
A TUSC Presentation
160

Rac Tuning Sig 2004

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Rac Tuning Sig 2004

Uploaded by

Copyright:

Available Formats

Oracle RAC SIG 2004

Performance Monitoring & Tuning for RAC

Overview of Oracle9i RAC

Oracle9i Database Clusters

Real Application Clusters

E-Business Suite Scalability with Oracle9i RAC

6,496 5,433 4,368* 2,296*

4,000 3,000 2,000 1,000 0

Analysis of Performance Issues

Normal Database Tuning and Monitoring

You can begin tuning RAC

Tuning the RAC Cluster Interconnect

Tuning the RAC Cluster Interconnect

Global Cache Services (GCS) Waits

buffer busy global CR

Waits on a consistent read via the global cache.

Tuning the RAC Cluster Interconnect

Tuning the RAC Cluster Interconnect

Tuning the RAC Cluster Interconnect

Tuning the RAC Cluster Interconnect

Tuning the RAC Cluster Interconnect

The output will be similar to:

Tuning the RAC Cluster Interconnect

3) Modify the object to reduce the chances for application contention.

Tuning the RAC Cluster Interconnect CR Block Transfer Time

global cache cr block receive time global cache cr blocks received

Tuning the RAC Cluster Interconnect CR Block Transfer Time

Use a self-join query on GV$SYSSTAT to compute this ratio:

FROM WHERE AND AND

Tuning the RAC Cluster Interconnect CR Block Transfer Time

Use OS commands to verify cluster interconnects are functioning correctly.

Tuning the RAC Cluster Interconnect CR Block Service Time

Tuning the RAC Cluster Interconnect CR Block Service Time

Query GV$SYSSTAT to determine average service times by instance:

Tuning the RAC Cluster Interconnect CR Block Service Time

Instance 1 is on a faster node and serves blocks faster.

The service time TO Instance 2 is shorter!

Tuning the RAC Cluster Interconnect CR Block Service Time

Tuning the RAC Cluster Interconnect OS Troubleshooting Commands

Monitor cluster interconnects using OS commands to find:

Tuning the RAC Cluster Interconnect OS Troubleshooting Commands

Useful OS commands in a Sun Solaris environment to aid in your research are:

Tuning the RAC Cluster Interconnect Your Analysis Toolkit

RACDIAG.SQL is an Oracle-provided script that can also help with troubleshooting.

Tuning the RAC Cluster Interconnect Undesirable Statistics

Tuning the RAC Cluster Interconnect Undesirable Statistics

Instance 2 is experiencing some lost blocks.

Tuning the RAC Cluster Interconnect Undesirable Statistics

Take a closer look to see what is causing the lost blocks:

Tuning the RAC Cluster Interconnect Undesirable Statistics

Tuning the RAC Cluster Interconnect Current Block Transfer Statistics

Tuning the RAC Cluster Interconnect Current Block Transfer Statistics

FROM WHERE AND AND

global cache block pin time

global cache block send time

To calculate current block service time, query GV$SYSSTAT:

The output from the query may look like this:

global cache get time

Interpreting the convert and get statistics

Tuning the RAC Cluster Interconnect Other Wait Events

Tuning the RAC Cluster Interconnect Other Wait Events

Event: global cache open s global cache open x

Tuning the RAC Cluster Interconnect Other Wait Events

Event: global cache null to s global cache null to x